Please create an account to participate in the Slashdot moderation system

 


Forgot your password?
Close
typodupeerror
×
Social Networks

Ask Slashdot: Best Practices For Using a Reputation Engine To Rate Information? 100

Posted by timothy from the well-if-grantrobertson-says-it dept.
GrantRobertson writes: For my graduate project, I am considering developing a web engine designed around sharing and organizing actual information in a way that people would actually like to and easily be able to use it. Unlike a wiki, the information will be much more granular with lots more metadata and organization. Unlike a web forum, the information will be be organized rather than dispersed throughout thousands of random posts, with little room for dominant personalities to take over. While I like Stack Overflow, I am planning far more structure. While I enjoy the entertaining tangents on Slashdot, I don't want those to take over sites created using my engine. Naturally, there must be some way to prevent armies of bots or just legions of jerks from derailing web sites created using this engine. Given that, what would you say are some good rules to include in the reputation engine for such a site. What kinds of algorithms have you found to be most beneficial to the propagation and spread of actual knowledge. What would you like to see and what have you found to be dismal failures?
This discussion has been archived. No new comments can be posted.

Ask Slashdot: Best Practices For Using a Reputation Engine To Rate Information? More

Ask Slashdot: Best Practices For Using a Reputation Engine To Rate Information?

Comments Filter:

  • You mean to tell me (Score:4, Insightful)

    by Anonymous Coward on Thursday December 24, 2015 @04:07PM (#51179557)
    you are counting on Slashdot to do your graduate project for you? That is a horrible idea in so many ways...

    • Re:You mean to tell me (Score:5, Insightful)

      by ShanghaiBill ( 739463 ) on Thursday December 24, 2015 @04:21PM (#51179607)

      you are counting on Slashdot to do your graduate project for you?

      No, the submitter is just asking for ideas. 99% of the work is the implementation, and the submitter is not asking for any help with that.

      My 2 cents: The two problems to avoid are: 1) Groupthink, where dissenting opinion are drowned out or ignored. 2) Onerous or arbitrary rules that drive away experts, so you are left with only clueless idiots commiserating with each other (example: answers.yahoo.com).

      • No, the submitter is just asking for ideas. 99% of the work is the implementation, and the submitter is not asking for any help with that.

        So then why don't they work that out with their thesis advisor? That's the entire reason you have one.

        Secondly, no, 99% of the work in a thesis is not implementation unless you're getting a degree from a shit school and you have a rubber-stamp committee. The vast majority of your thesis is in the original research you did and writing down your findings from that research. The implementation is just a prototype to show off a working example of your ideas.

        • Re: (Score:1, Interesting)

          by Anonymous Coward

          He's probably seeking ideas from many different sources. Nothing wrong with that.

          On a related note, you are exactly they type of poster his project would automatically mod down to oblivion; I hope he succeeds.

        • The vast majority of your thesis is in the original research you did and writing down your findings from that research. The implementation is just a prototype to show off a working example of your ideas.

          That is a good point. I struggle with that issue constantly. I am mostly an idea guy, but academia is all about the research. So I constantly have to figure out how to couch my ideas in terms of the research that could be done around them. Not just, "Hey, here is my great idea!" but, "How can this idea be exactly how much better than some past idea, along some measurable vector?" It is a whole different mindset.

        • why would his thesis advisor have more knowledge of attempts at reputation systems than the /. readership? Most of what's tried in industry is buried next to the bodies behind the data center, not in research papers. Field research is usually more valuable than literature surveys.

      • The two problems to avoid are: 1) Groupthink, where dissenting opinion are drowned out or ignored. 2) Onerous or arbitrary rules that drive away experts, so you are left with only clueless idiots commiserating with each other (example: answers.yahoo.com).

        Good points. Groupthink is one of the main things I am trying to avoid. Groupthink can so very often be wrong.

        Though I have complained about such arbitrary rules myself, I hadn't thought about that being a problem in any system I created. (I know, we never think we will make the same mistakes we complain about ourselves, but it can happen.) I will remember to keep the actual use of the system as simple as possible.

        • I just want to point out that the group here seems to be in consensus that groupthink is a problem. Or to put it another way, the groupthink is that groupthink is a problem to be avoided. Except in this case groupthink is probably right.

          Now if that makes sense to you, you should realize that i have no idea what a reputation engine is (other than a means to validate information) or how that could be implemented without authoritative answers around the latest information). I'm not sure it could be reliably d

      • you are counting on Slashdot to do your graduate project for you?

        No, the submitter is just asking for ideas. 99% of the work is the implementation, and the submitter is not asking for any help with that.

        Betting that's coming as a follow-up article Real Soon Now :)

      • you are counting on Slashdot to do your graduate project for you?

        No, the submitter is just asking for ideas. 99% of the work is the implementation, and the submitter is not asking for any help with that.

        My 2 cents: The two problems to avoid are: 1) Groupthink, where dissenting opinion are drowned out or ignored. 2) Onerous or arbitrary rules that drive away experts, so you are left with only clueless idiots commiserating with each other (example: answers.yahoo.com).

        Yes, but that 1% of inspiration is what the thesis markers are looking for, its what distinguishes one thesis from another.

        • Well, I can stand on the shoulders of giants. Or I can stand on the shoulders of two dozen trolls. What I'm doing here is trying to stand with one foot on the giants and one foot on the trolls. It'll be a precarious balance, I know. But I believe in Science, rather than Academia. I believe the best ideas can come from anywhere and everywhere. So, if I have to consort with trolls to stand with giants, so be it.

          (I know. That sounds insanely egotistical. If I am ultimately successful, it will be a profound quo

    • Re:You mean to tell me (Score:4, Insightful)

      by Mikkeles ( 698461 ) on Thursday December 24, 2015 @05:27PM (#51179849)

      No. He is asking users for features and characteristics that said users would find advantageous for a web engine that accumulates and organizes web data.

      <snark>Something I wish more programmers would do.</snark>

      • No. He is asking users for features and characteristics that said users would find advantageous for a web engine that accumulates and organizes web data.

        <snark>Something I wish more programmers would do.</snark>

        So you want to re-invent wikipedia?

      • No. He is asking users for features and characteristics that said users would find advantageous for a web engine that accumulates and organizes web data.

        Users... aren't they those things that bitch about how Open Source happens to work, and then don't contribute patches back to address those complaints?

        (NB: Not precisely my view, but it's going to be the typical view of most people).

  • Impartial referees (Score:4, Insightful)

    by forgottenusername ( 1495209 ) on Thursday December 24, 2015 @04:10PM (#51179565)

    Pretty much any set of algos is going to be easily defeated by humans trolling and no system is going to be anything near perfect. My thoughts;

    1) Create a small set of simple, concise rules that are inviolate
    2) Have a system so people can mark submissions as good (no rules broken/useful) or bad(rules broken)
    3) Have your referees do nothing but determine if that submission is breaking one of your rules
    4) Based your user trust as a derivative as how the user voted compared to what the referee votes

    The theory is any controversial submission is going to get flagged & referees attention. Their job is limited in scope to just determining if the post breaks the site rules or not, nothing to do with quality / content / opinion. If users are trying to game the system their votes are going to conflict with the referees so their user trust is going to go down, whereas if people agree their trust is going to go up.

    Eventually you'll have a group of users that you can generally trust to do the right thing so you can weight their actions accordingly.

    Obviously there are some weaknesses;

    - Referees are pretty much god (that's why the scope of their power is extremely narrow and simple)
    - You can end up with hive mind (though you can combat that if enough trusted users conflict with other trusted users). I'd argue it's a way better protection than pure crowdsourcing ala reddit where the demographics crush submissions into hivemind

    Just tossing that out there off the top of my head. It's not something to replace automated reputation management, just something augment it and limit some of the abuse.

    • Sounds like Stack.

      • I appreciate the voting system used by Stack Exchange, though my system would be more organized and said organization would also be crowd-sourced. So, I am thinking I would need some variation on the system used by Stack Exchange.

    • Thanks. This is just the kind of helpful comment I was hoping for.

      • I agree with GP, and I would also keep user names/ids and reputation scores hidden so people are less inclined to farm their scores or band together to level up. I may be able to indirectly see the effects of my reputation, but I can't be sure, and it won't be easy for me to demonstrate it to other people.

        BTW I think Google Search is in effect a pretty good reputation engine, it's just not apparent.

        Also, for me, the Slashdot system of moderation and metamoderation works well with the set of custom fi

        • Yes, I had been thinking of showing reputation scores going from 1 - 10 but allowing the internal, hidden, weights to go much higher. Research has shown it is effective to encourage novices, as in rewarding them with increasing reputation scores, but that has diminishing returns. Once people become more skilled, they respond better to specific constructive criticisms. So, I was thinking that, if someone wanted to downvote a contribution, that should have to give a specific reason that is shown only to the o

  • Homework question. lol (Score:1)

    by Anonymous Coward

    Should someone inform their thesis advisor that they are getting others to do his work for him?

  • Ummm... (Score:4, Insightful)

    by Lunix Nutcase ( 1092239 ) on Thursday December 24, 2015 @04:17PM (#51179599)

    Isn't the whole point of thesis work that you find some novel solution to a problem through your own research not enlisting others to do it for you?

    • Isn't the whole point of thesis work that you find some novel solution to a problem through your own research not enlisting others to do it for you?

      No.

    • Re:Ummm... (Score:4, Insightful)

      by ljw1004 ( 764174 ) on Thursday December 24, 2015 @05:40PM (#51179919)

      Isn't the whole point of thesis work that you find some novel solution to a problem through your own research not enlisting others to do it for you?

      That's maybe 5% of thesis work. Another 20% is the grunt-work to investigate the phenomena and gather up examples and counter-examples. Another 75% is getting a good understanding of the field and the existing state of the art.

      I think the poster has picked a good place (here on slashdot) as part of building up that other 95%.

    • In an earlier age, yes. But today the internet makes it easy to ask other people to do your homework for you. And there are tons of people out there just bursting with helpfulness who will actually do it. Doesn't help the student learn...but that's not the point. The point is to get the diploma, you can't get ahead without one.

    • In practice OP is not going to get actual verified solutions but a bunch of half-baked ideas that merely confirm the respondents' priors. OP can then analyze each of the responses for the claimed gain. That analysis constitutes the real research.

  • "Unlike a wiki, the information will be much more granular with lots more metadata and organization."
    Pretty sure the ideals behind a semantic web were supposed to cover this part. Never really took off though because, I think, people are to lazy to sort data to that degree of detail and the algorithms necessary to process and categorize human text with that level of granularity seem to be very hard to make.

  • Look into mTurk (Mechanical Turk). Amazon doesn't provide a reputation engine, but anyone who posts any significant number of jobs there has some kind of version of it. I worked for several years on a project that integrated with mTurk and had its own reputation engine. There are a lot of gotchas where people try to game the system. It isn't a simple answer and depending on the situation I don't believe there is a one solution for all situations. Bill

  • the only reputation system that will ever beat legions of jerks will have to be able to determine if the information itself is correct. when dealing with jerks, you need to remember they are humans, the most cunning and devious of superpredators. [fbcoolcovers.com] jerks will build a good reputation by giving good answers just to destroy the reputations of others or build up reputation of jerks that give bad answers. no system you come up with will be infallible.

  • ...the power of jerks!

  • sounds like OneModel; maybe you can start there (Score:3, Informative)

    by lcall ( 143264 ) on Thursday December 24, 2015 @05:22PM (#51179829) Homepage

    Sounds like what I'm trying to do here (AGPL): OneModel [onemodel.org].

    It doesn't have all the features, but what you describe is partly there, or planned for the future, though for now it's in the form of a text-only UI and you have to install postgres. The UI is something like a mix of git's "commit --interactive" and gopher (remember that, anyone?), but it is very efficient if you just read the screen and are a touch typist. Probably currently most suitable for someone who now uses emacs org-mode, or collapsible outlines of any kind, but wants to handle richer kinds of information (eg, GTD...) and a more task-specific UI.

    It's what I use as my own personal organizer and knowledge manager, but ~"sharing" features for collaboration, including reputation and others, are on the wish/plan list. Feel free to use it as a starting point, or join the list for discussion. I was hoping to get the web site updated with a later binary and an enhancement, and much more information on my future plans, by roughly next week. It still lacks a convenient installer but the INSTALLING file in github is current.

    If interested you could always get on the announcements list for when I add features. My health isn't great at the moment but I hope to be able to sell binaries or installers in the future for part-time income or the like. Patches or discussion on the list are welcome. I have been thinking hard about this since about 2000 and am glad to finally have something others can use, though the potential audience will be larger once there are better installers and other needed features, UIs etc.

    • Feature-wise, OM is more of what you describe for the structure of the information, right now as a personal organizer. It doesn't address the reputation question but that is definitely something I've been thinking about and seems like a fit, long-term. Nearer-term is being able to integrate data across individual instances, with reputation being a closely-related issue.

      • I've tried really hard to find any information that would explain what makes OneModel different than "wiki in list format" (not trying to be snarky here), but I've ended up empty handed... could you please explain further? Is it really a custom UI with a wiki backend (conceptually speaking)?

        • Re: (Score:1)

          by lcall ( 143264 )

          Short-term, you're not too far off. Medium- and long-term, the underlying structure and plan is for it to model *knowledge*, as opposed to tons of words. I think knowledge at an ~"atomic level" is more about numbers and relationships than about tons of words. I.e., if an entity being modeled (a "knowledge unit" if we use the expression) is called a different thing in another language, it's still the same thing, just has a different name (though granted, contexutally-driven names for things isn't there ye

        • Re: (Score:1)

          by lcall ( 143264 )

          Also, the UI is very efficient for brainstorming then rearranging things (including an improvement I'm about 2/3 done with). And because it is structured underneath, one can "model" one's knowledge to represent what "is". One of the nearer-future features I hope for will use that structure to allow sharing between models: if one person develops info in a structure and makes it available, others could monitor it for changes (subscribe in a way if desired), and copy or link to pieces of the model. But not o

          • Interesting. You need to be careful with the UI, though. It might be easy to express one's thoughts as a knowledge-graph, because you're familiar with each node individually. But for others to traverse that graph the structure needs to be much more organized. You can easily get "lost" in cliques that hide content not very well connected. It's much easier for a newcomer to traverse a tree, and there is the ages-old HMI rule of "no more than 3 levels and 7 items per level".

            About computation, you might find th

            • Re: (Score:1)

              by lcall ( 143264 )

              Thanks; I've noted the link for future use. In the docs I'll try to show various hiarchical / top-down ways of organizing information. There's also a search feature in the current desktop version. But I don't know of a way to capture growing amounts of arbitrary knowledge with the 3 level, 7 items per level rule. Hopefully each level can try to comply... maybe something like wikipedia does.

              A user can make their own structure for their own or others' data, led by their own "top-level-memorable" entities.

        • Re: (Score:1)

          by lcall ( 143264 )

          Here's another way to look at it.

          When we create software, it often involves having a developer create a custom object model for the need at hand. This is expensive.

          So for manipulating knowledge in a generic way, I have hoped this system (or something like it) can be used to create an object model for any kind of knowledge, on the fly, as a side-effect of using the system, about as easily as one might use a word processor.

          If you look at some physical object near you now, what you *know* about it can be model

    • I just realized there's a startup bug for first-time users; I'll try to fix that & post back here in a few hours or later tonight. (sorry for not being better prepared but this seemed like an opportunity to share something useful.)

      • Re: (Score:1)

        by lcall ( 143264 )

        (Now I can't seem to reproduce it. Seems to be working. Except that it didn't find the license file since i didn't put it in the .jar or look there yet, but one can press Enter through that and get to the data creation just fine.)

    • Thanks for the link. I will look into your system. I would be more interested in the data model itself, rather than the user interface. I'll let you know if I decide to steal any ideas. ;^)

  • More structure? (Score:3)

    by YrWrstNtmr ( 564987 ) on Thursday December 24, 2015 @05:37PM (#51179903)
    While I like Stack Overflow, I am planning far more structure.

    More? Good grief. SO is already bad enough. Anything 'more' will simply chase users away, if they ever go there in the first place.

  • pay someone (Score:2, Insightful)

    by superwiz ( 655733 )
    I have a brilliant answer to your question. But it seems like you want it answered for a big shiny price of "free". I'll keep it to myself. Oh, and if you are thinking of having a contest and hope to get my idea without actually paying for it (and no, having a contest is not it), you can forget about it. I won't submit to any such contest. If you want data analytics ideas start paying people who spend time of their lives learning how to do data analytics.

  • Allow multiple self selected groups to provide ratings and let the reader select which rating system to use. The opinions of some raters is more important than other raters.

    • Oh, that is a REALLY good idea. I had planned to have multiple hierarchical organization systems in the knowledge portion of the system, but I hadn't thought of doing the same in the reputation engine. If one really expands on this idea, one could consider the possibility of multiple, independent, competing reputation "services" that people sign up for. Kind of like there are multiple different search engines for the internet. And like there are componentized discussion engines, such as Disqus, that can be

  • Well, we’ve all known for some time that Slashdot could stand to have a better reputation engine of some sort, just to filter out most of the kinds of comments I’m getting here. Be that as it may, I will try to have a conversation with the actual thinking individuals who still come here, over the noise of the trolls.

    In answer to some of the protests:

    If anyone thinks a few opinions, randomly thrown around, here on Slashdot can, in any way, shape, or form, constitute the bulk of the work for a gra

  • At kr5ddit.com [kr5ddit.com].

    Instead of one user one vote, we have one kr5ddit, one vote.

    We use kr5dditz, which are like karma, to determine how much you can moderate. You earn kr5dditz by moderating and and by being moderated. You can also buy and sell kr5dditz on our exchange for bitcoin.

    I believe that this system should be robust in the face of sock puppets and bad actors... but time will tell.

    Anyway, feel free to pop over and register [kr5ddit.com], and talk with me about how it works. The site is under development, so lots of stu

    • I seriously recommend you check us out... I have put the last 4 months of my time into this project... I have a background in software engineering, and have a pretty good grasp on the fundamentals of micro-economics.

      I have tried to apply principals from micro-economics to the site... Realising that posting and moderation are both really externalities... meaning that they suffer from tragedy of the commons type problems... and the solutions to externalities is (pigovian) taxes and subsidies... So, actually,

    • I will definitely look into your system. The only part I don't like is being able to buy influence. But earning influence/karma in one place and spending it in another is an interesting idea. Though it does introduce the possibility af "karma mills" where a site is created just to allow people to easily build karma or to sell karma on the black market.

      I think the karma should always be earned.

      • The trading karma for bitcoin isn't necessary... Though it does allow people to make a little bit of money from the content they create, and it also allows those with unpopular opinions the ability to purchase their right to have those opinions heard.

        We only allow trading of kr5dditz (karma) for bitcoin... So, the conversion of one type of utility to another... there's nothing that says they have to build sites as karma mills... they can buy bitcoin for cash, and then kr5dditz for bitcoin directly... Though

  • For older people who do know this song and young'uns who need to become acquainted with it, and, indeed, the whole of his canon: https://youtu.be/gXlfXirQF3A [youtu.be] Happy Whatever.
  • "For my graduate project, I am considering developing a web engine designed around sharing and organizing actual information in a way that people would actually like to and easily be able to use it".

    It depends on the quality of the posters, you should aim for something like news.ycombinator.com [ycombinator.com]

  • DICE edited my headline. (Score:3)

    by GrantRobertson ( 973370 ) on Friday December 25, 2015 @09:39AM (#51181785) Homepage Journal

    I would like to point out that DICE edited my headline, which was originally, "Reputation Engine - Best Practices for Information-Based Site?" The existing headline makes it appear as if I am trying to use the reputation engine to rate the actual information. Instead, I merely want the reputation engine to cut down on the number of jerks on the site and reduce the influence of trolls, bots, and crusading armies. Once that is accomplished, I trust the "good" contributors to provide good and relatively accurate content by working together and collaborating. I do not expect any reputation engine to get to some ethereal "Truth."

Slashdot Top Deals

He has not acquired a fortune; the fortune has acquired him. -- Bion

Close