Ask Slashdot: Best Practices For Using a Reputation Engine To Rate Information? 100
GrantRobertson writes: For my graduate project, I am considering developing a web engine designed around sharing and organizing actual information in a way that people would actually like to and easily be able to use it. Unlike a wiki, the information will be much more granular with lots more metadata and organization. Unlike a web forum, the information will be be organized rather than dispersed throughout thousands of random posts, with little room for dominant personalities to take over. While I like Stack Overflow, I am planning far more structure. While I enjoy the entertaining tangents on Slashdot, I don't want those to take over sites created using my engine. Naturally, there must be some way to prevent armies of bots or just legions of jerks from derailing web sites created using this engine. Given that, what would you say are some good rules to include in the reputation engine for such a site. What kinds of algorithms have you found to be most beneficial to the propagation and spread of actual knowledge. What would you like to see and what have you found to be dismal failures?
You mean to tell me (Score:4, Insightful)
Re:You mean to tell me (Score:5, Insightful)
you are counting on Slashdot to do your graduate project for you?
No, the submitter is just asking for ideas. 99% of the work is the implementation, and the submitter is not asking for any help with that.
My 2 cents: The two problems to avoid are: 1) Groupthink, where dissenting opinion are drowned out or ignored. 2) Onerous or arbitrary rules that drive away experts, so you are left with only clueless idiots commiserating with each other (example: answers.yahoo.com).
Re: (Score:3)
No, the submitter is just asking for ideas. 99% of the work is the implementation, and the submitter is not asking for any help with that.
So then why don't they work that out with their thesis advisor? That's the entire reason you have one.
Secondly, no, 99% of the work in a thesis is not implementation unless you're getting a degree from a shit school and you have a rubber-stamp committee. The vast majority of your thesis is in the original research you did and writing down your findings from that research. The implementation is just a prototype to show off a working example of your ideas.
Re: (Score:2, Interesting)
How is it tangent? If he can't even come up with his own ideas even after talking to his advisors than thesis work is probably beyond his level. And again, his very questions are what he should be answering himself with his own research. That's the whole point behind a grad thesis to begin with.
Re: (Score:2)
Meh. I've seen better, and much funnier before.
Seriously. Where is my moo cow? It is not a real Slashdot thread without the moo cow. Of course now that I have asked for it, I have probably disqualified myself from moocowdom.
Re: (Score:1, Interesting)
He's probably seeking ideas from many different sources. Nothing wrong with that.
On a related note, you are exactly they type of poster his project would automatically mod down to oblivion; I hope he succeeds.
Re: (Score:2)
The vast majority of your thesis is in the original research you did and writing down your findings from that research. The implementation is just a prototype to show off a working example of your ideas.
That is a good point. I struggle with that issue constantly. I am mostly an idea guy, but academia is all about the research. So I constantly have to figure out how to couch my ideas in terms of the research that could be done around them. Not just, "Hey, here is my great idea!" but, "How can this idea be exactly how much better than some past idea, along some measurable vector?" It is a whole different mindset.
Re: You mean to tell me (Score:3)
why would his thesis advisor have more knowledge of attempts at reputation systems than the /. readership? Most of what's tried in industry is buried next to the bodies behind the data center, not in research papers. Field research is usually more valuable than literature surveys.
Re: You mean to tell me (Score:2)
Bingo!
Re: (Score:1)
I'd mod you up, but then I wouldn't be able to comment in my own thread. So...
Re: (Score:2)
The two problems to avoid are: 1) Groupthink, where dissenting opinion are drowned out or ignored. 2) Onerous or arbitrary rules that drive away experts, so you are left with only clueless idiots commiserating with each other (example: answers.yahoo.com).
Good points. Groupthink is one of the main things I am trying to avoid. Groupthink can so very often be wrong.
Though I have complained about such arbitrary rules myself, I hadn't thought about that being a problem in any system I created. (I know, we never think we will make the same mistakes we complain about ourselves, but it can happen.) I will remember to keep the actual use of the system as simple as possible.
Re: (Score:2)
I just want to point out that the group here seems to be in consensus that groupthink is a problem. Or to put it another way, the groupthink is that groupthink is a problem to be avoided. Except in this case groupthink is probably right.
Now if that makes sense to you, you should realize that i have no idea what a reputation engine is (other than a means to validate information) or how that could be implemented without authoritative answers around the latest information). I'm not sure it could be reliably d
Re: (Score:2)
you are counting on Slashdot to do your graduate project for you?
No, the submitter is just asking for ideas. 99% of the work is the implementation, and the submitter is not asking for any help with that.
Betting that's coming as a follow-up article Real Soon Now :)
Re: (Score:2)
you are counting on Slashdot to do your graduate project for you?
No, the submitter is just asking for ideas. 99% of the work is the implementation, and the submitter is not asking for any help with that.
My 2 cents: The two problems to avoid are: 1) Groupthink, where dissenting opinion are drowned out or ignored. 2) Onerous or arbitrary rules that drive away experts, so you are left with only clueless idiots commiserating with each other (example: answers.yahoo.com).
Yes, but that 1% of inspiration is what the thesis markers are looking for, its what distinguishes one thesis from another.
Re: You mean to tell me (Score:2)
Well, I can stand on the shoulders of giants. Or I can stand on the shoulders of two dozen trolls. What I'm doing here is trying to stand with one foot on the giants and one foot on the trolls. It'll be a precarious balance, I know. But I believe in Science, rather than Academia. I believe the best ideas can come from anywhere and everywhere. So, if I have to consort with trolls to stand with giants, so be it.
(I know. That sounds insanely egotistical. If I am ultimately successful, it will be a profound quo
Re:You mean to tell me (Score:4, Insightful)
No. He is asking users for features and characteristics that said users would find advantageous for a web engine that accumulates and organizes web data.
<snark>Something I wish more programmers would do.</snark>
Re: (Score:2)
No. He is asking users for features and characteristics that said users would find advantageous for a web engine that accumulates and organizes web data.
<snark>Something I wish more programmers would do.</snark>
So you want to re-invent wikipedia?
Users... (Score:2)
No. He is asking users for features and characteristics that said users would find advantageous for a web engine that accumulates and organizes web data.
Users... aren't they those things that bitch about how Open Source happens to work, and then don't contribute patches back to address those complaints?
(NB: Not precisely my view, but it's going to be the typical view of most people).
Re: (Score:2)
While I enjoy the entertaining tangents on Slashdot,
Not only is he coming to Slashdot, he is insulting the people posting here.
If you want favourable responses and constructive conversation, don't start out by calling your audience/source of feedback a bunch of jerks.
Stay in school, you fail at life.
Also someone who fails to notice that many times, those "entertaining tangents" are laden with information and more interesting than the story itself. Otherwise, why would anyone read the comments (especially at -1, where there's gold among the turds)?
Impartial referees (Score:4, Insightful)
Pretty much any set of algos is going to be easily defeated by humans trolling and no system is going to be anything near perfect. My thoughts;
1) Create a small set of simple, concise rules that are inviolate
2) Have a system so people can mark submissions as good (no rules broken/useful) or bad(rules broken)
3) Have your referees do nothing but determine if that submission is breaking one of your rules
4) Based your user trust as a derivative as how the user voted compared to what the referee votes
The theory is any controversial submission is going to get flagged & referees attention. Their job is limited in scope to just determining if the post breaks the site rules or not, nothing to do with quality / content / opinion. If users are trying to game the system their votes are going to conflict with the referees so their user trust is going to go down, whereas if people agree their trust is going to go up.
Eventually you'll have a group of users that you can generally trust to do the right thing so you can weight their actions accordingly.
Obviously there are some weaknesses;
- Referees are pretty much god (that's why the scope of their power is extremely narrow and simple)
- You can end up with hive mind (though you can combat that if enough trusted users conflict with other trusted users). I'd argue it's a way better protection than pure crowdsourcing ala reddit where the demographics crush submissions into hivemind
Just tossing that out there off the top of my head. It's not something to replace automated reputation management, just something augment it and limit some of the abuse.
Re: Impartial referees (Score:2)
Re: (Score:2)
I appreciate the voting system used by Stack Exchange, though my system would be more organized and said organization would also be crowd-sourced. So, I am thinking I would need some variation on the system used by Stack Exchange.
Re: (Score:2)
Thanks. This is just the kind of helpful comment I was hoping for.
Re: (Score:2)
BTW I think Google Search is in effect a pretty good reputation engine, it's just not apparent.
Also, for me, the Slashdot system of moderation and metamoderation works well with the set of custom fi
Carrots and sticks. (Score:2)
Yes, I had been thinking of showing reputation scores going from 1 - 10 but allowing the internal, hidden, weights to go much higher. Research has shown it is effective to encourage novices, as in rewarding them with increasing reputation scores, but that has diminishing returns. Once people become more skilled, they respond better to specific constructive criticisms. So, I was thinking that, if someone wanted to downvote a contribution, that should have to give a specific reason that is shown only to the o
Homework question. lol (Score:1)
Should someone inform their thesis advisor that they are getting others to do his work for him?
Ummm... (Score:4, Insightful)
Isn't the whole point of thesis work that you find some novel solution to a problem through your own research not enlisting others to do it for you?
Re: (Score:1)
Isn't the whole point of thesis work that you find some novel solution to a problem through your own research not enlisting others to do it for you?
No.
Re: (Score:3)
Then you went to a diploma mill not a real university.
Re: (Score:2)
I attended Reed College in Portland (http://www.reed.edu/) and obtained my Masters at the University of Oregon.
I assume you attended the University of Phoenix...
Re: (Score:2)
Reed College is great if you're interested in starting a heroin dependency.
Re: (Score:2)
Hopefully his work will carry over how on Stack Overflow they don't fall for these "Do my homework" questions.
Re:Ummm... (Score:4, Insightful)
Isn't the whole point of thesis work that you find some novel solution to a problem through your own research not enlisting others to do it for you?
That's maybe 5% of thesis work. Another 20% is the grunt-work to investigate the phenomena and gather up examples and counter-examples. Another 75% is getting a good understanding of the field and the existing state of the art.
I think the poster has picked a good place (here on slashdot) as part of building up that other 95%.
Re: (Score:1)
You are correct. Thank you.
Re: (Score:2)
Re: (Score:2)
In practice OP is not going to get actual verified solutions but a bunch of half-baked ideas that merely confirm the respondents' priors. OP can then analyze each of the responses for the claimed gain. That analysis constitutes the real research.
Re: (Score:2)
Yup.
Re: (Score:2)
See, now here is some of that gold spoken of earlier. While I am somewhat familiar with deep learning, I hadn't thought of using it to mine trust information out of the entire database of comment and voting information. Possibly across an entire swath of associated sites.
Remember, this fuzzy stuff is not my strong suit. My real project is organizing hard information into hierarchies. Kinda the opposite of sussing out the real intent of something as mushy as anonymous internet users. I was just going to try
Semantic web? (Score:2)
"Unlike a wiki, the information will be much more granular with lots more metadata and organization."
Pretty sure the ideals behind a semantic web were supposed to cover this part. Never really took off though because, I think, people are to lazy to sort data to that degree of detail and the algorithms necessary to process and categorize human text with that level of granularity seem to be very hard to make.
Re: Semantic web? (Score:2)
Re: (Score:2)
Actually, it did make sense. Still does. Most just can't make sense OF it, and it doesn't make it easy to sell eyeballs. However, that is yet a different project, which I may or may not have time to work on in my lifetime. I call it "Web 0.0"
Reputations... (Score:2)
it will at least have to pass the turring test (Score:2)
the only reputation system that will ever beat legions of jerks will have to be able to determine if the information itself is correct. when dealing with jerks, you need to remember they are humans, the most cunning and devious of superpredators. [fbcoolcovers.com] jerks will build a good reputation by giving good answers just to destroy the reputations of others or build up reputation of jerks that give bad answers. no system you come up with will be infallible.
Don't underestimate... (Score:1)
sounds like OneModel; maybe you can start there (Score:3, Informative)
Sounds like what I'm trying to do here (AGPL): OneModel [onemodel.org].
It doesn't have all the features, but what you describe is partly there, or planned for the future, though for now it's in the form of a text-only UI and you have to install postgres. The UI is something like a mix of git's "commit --interactive" and gopher (remember that, anyone?), but it is very efficient if you just read the screen and are a touch typist. Probably currently most suitable for someone who now uses emacs org-mode, or collapsible outlines of any kind, but wants to handle richer kinds of information (eg, GTD...) and a more task-specific UI.
It's what I use as my own personal organizer and knowledge manager, but ~"sharing" features for collaboration, including reputation and others, are on the wish/plan list. Feel free to use it as a starting point, or join the list for discussion. I was hoping to get the web site updated with a later binary and an enhancement, and much more information on my future plans, by roughly next week. It still lacks a convenient installer but the INSTALLING file in github is current.
If interested you could always get on the announcements list for when I add features. My health isn't great at the moment but I hope to be able to sell binaries or installers in the future for part-time income or the like. Patches or discussion on the list are welcome. I have been thinking hard about this since about 2000 and am glad to finally have something others can use, though the potential audience will be larger once there are better installers and other needed features, UIs etc.
clarification (Score:1)
Feature-wise, OM is more of what you describe for the structure of the information, right now as a personal organizer. It doesn't address the reputation question but that is definitely something I've been thinking about and seems like a fit, long-term. Nearer-term is being able to integrate data across individual instances, with reputation being a closely-related issue.
Re: (Score:2)
I've tried really hard to find any information that would explain what makes OneModel different than "wiki in list format" (not trying to be snarky here), but I've ended up empty handed... could you please explain further? Is it really a custom UI with a wiki backend (conceptually speaking)?
Re: (Score:1)
Short-term, you're not too far off. Medium- and long-term, the underlying structure and plan is for it to model *knowledge*, as opposed to tons of words. I think knowledge at an ~"atomic level" is more about numbers and relationships than about tons of words. I.e., if an entity being modeled (a "knowledge unit" if we use the expression) is called a different thing in another language, it's still the same thing, just has a different name (though granted, contexutally-driven names for things isn't there ye
Re: (Score:1)
Also, the UI is very efficient for brainstorming then rearranging things (including an improvement I'm about 2/3 done with). And because it is structured underneath, one can "model" one's knowledge to represent what "is". One of the nearer-future features I hope for will use that structure to allow sharing between models: if one person develops info in a structure and makes it available, others could monitor it for changes (subscribe in a way if desired), and copy or link to pieces of the model. But not o
Re: (Score:2)
Interesting. You need to be careful with the UI, though. It might be easy to express one's thoughts as a knowledge-graph, because you're familiar with each node individually. But for others to traverse that graph the structure needs to be much more organized. You can easily get "lost" in cliques that hide content not very well connected. It's much easier for a newcomer to traverse a tree, and there is the ages-old HMI rule of "no more than 3 levels and 7 items per level".
About computation, you might find th
Re: (Score:1)
Thanks; I've noted the link for future use. In the docs I'll try to show various hiarchical / top-down ways of organizing information. There's also a search feature in the current desktop version. But I don't know of a way to capture growing amounts of arbitrary knowledge with the 3 level, 7 items per level rule. Hopefully each level can try to comply... maybe something like wikipedia does.
A user can make their own structure for their own or others' data, led by their own "top-level-memorable" entities.
Re: (Score:1)
Here's another way to look at it.
When we create software, it often involves having a developer create a custom object model for the need at hand. This is expensive.
So for manipulating knowledge in a generic way, I have hoped this system (or something like it) can be used to create an object model for any kind of knowledge, on the fly, as a side-effect of using the system, about as easily as one might use a word processor.
If you look at some physical object near you now, what you *know* about it can be model
bugfix coming (Score:1)
I just realized there's a startup bug for first-time users; I'll try to fix that & post back here in a few hours or later tonight. (sorry for not being better prepared but this seemed like an opportunity to share something useful.)
Re: (Score:1)
(Now I can't seem to reproduce it. Seems to be working. Except that it didn't find the license file since i didn't put it in the .jar or look there yet, but one can press Enter through that and get to the data creation just fine.)
Re: (Score:2)
Thanks for the link. I will look into your system. I would be more interested in the data model itself, rather than the user interface. I'll let you know if I decide to steal any ideas. ;^)
More structure? (Score:3)
More? Good grief. SO is already bad enough. Anything 'more' will simply chase users away, if they ever go there in the first place.
pay someone (Score:2, Insightful)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
multiple ratings (Score:2)
Allow multiple self selected groups to provide ratings and let the reader select which rating system to use. The opinions of some raters is more important than other raters.
Re: (Score:2)
Oh, that is a REALLY good idea. I had planned to have multiple hierarchical organization systems in the knowledge portion of the system, but I hadn't thought of doing the same in the reputation engine. If one really expands on this idea, one could consider the possibility of multiple, independent, competing reputation "services" that people sign up for. Kind of like there are multiple different search engines for the internet. And like there are componentized discussion engines, such as Disqus, that can be
Response From OP. (Score:2)
Well, we’ve all known for some time that Slashdot could stand to have a better reputation engine of some sort, just to filter out most of the kinds of comments I’m getting here. Be that as it may, I will try to have a conversation with the actual thinking individuals who still come here, over the noise of the trolls.
In answer to some of the protests:
If anyone thinks a few opinions, randomly thrown around, here on Slashdot can, in any way, shape, or form, constitute the bulk of the work for a gra
Re: (Score:2)
Yes, I need to read a lot more Clay Shirky. Though he does have his failings. I feel he tends to focus on the rhetoric espoused by the creators of a site rather than the actual mechanisms involved. He will talk about the rhetoric espoused by Site A and the results obtained, and then compare that to the rhetoric espoused by Site B and the results obtained. If on the internet no one knows if you are a dog, then it is even more true that on the internet no one cares about your rhetoric (at least in this contex
This is the problem I am trying to solve: (Score:2)
At kr5ddit.com [kr5ddit.com].
Instead of one user one vote, we have one kr5ddit, one vote.
We use kr5dditz, which are like karma, to determine how much you can moderate. You earn kr5dditz by moderating and and by being moderated. You can also buy and sell kr5dditz on our exchange for bitcoin.
I believe that this system should be robust in the face of sock puppets and bad actors... but time will tell.
Anyway, feel free to pop over and register [kr5ddit.com], and talk with me about how it works. The site is under development, so lots of stu
Re: (Score:2)
I seriously recommend you check us out... I have put the last 4 months of my time into this project... I have a background in software engineering, and have a pretty good grasp on the fundamentals of micro-economics.
I have tried to apply principals from micro-economics to the site... Realising that posting and moderation are both really externalities... meaning that they suffer from tragedy of the commons type problems... and the solutions to externalities is (pigovian) taxes and subsidies... So, actually,
Re: This is the problem I am trying to solve: (Score:2)
I will definitely look into your system. The only part I don't like is being able to buy influence. But earning influence/karma in one place and spending it in another is an interesting idea. Though it does introduce the possibility af "karma mills" where a site is created just to allow people to easily build karma or to sell karma on the black market.
I think the karma should always be earned.
Re: (Score:2)
The trading karma for bitcoin isn't necessary... Though it does allow people to make a little bit of money from the content they create, and it also allows those with unpopular opinions the ability to purchase their right to have those opinions heard.
We only allow trading of kr5dditz (karma) for bitcoin... So, the conversion of one type of utility to another... there's nothing that says they have to build sites as karma mills... they can buy bitcoin for cash, and then kr5dditz for bitcoin directly... Though
Nearly-Obligatory Tom Lehrer's 'Lobachevsky' (Score:2)
Re: (Score:2)
Awesome! Thanks.
Re: (Score:2)
Use news.ycombinator.com (Score:2)
It depends on the quality of the posters, you should aim for something like news.ycombinator.com [ycombinator.com]
DICE edited my headline. (Score:3)
I would like to point out that DICE edited my headline, which was originally, "Reputation Engine - Best Practices for Information-Based Site?" The existing headline makes it appear as if I am trying to use the reputation engine to rate the actual information. Instead, I merely want the reputation engine to cut down on the number of jerks on the site and reduce the influence of trolls, bots, and crusading armies. Once that is accomplished, I trust the "good" contributors to provide good and relatively accurate content by working together and collaborating. I do not expect any reputation engine to get to some ethereal "Truth."
Re: DICE edited my headline. (Score:2)
Good point. Thanks.
I guess where I am really going is a two level system. A reputation engine to rate the reliability and trustworthiness of contributers. Then a voting system for the content where the votes are weighted based on the reputation of the voter.
Re: Advogato (Score:2)
SWEET!!! This is exactly the kind of reference I was hoping to receive. All the trolls were worth this one reference. This site, and the project members themselves, will be a vast goldmine of information and potential collaboration. I don't know if I ever would have come across this on my own.
Thank you so much.
Grant