Ask Slashdot: How Does Your Team Track And Manage Bugs In Your Software? 189
Slashdot reader jb373 is a senior software engineer whose team's bug-tracking methodology is making it hard to track bugs.
My team uses agile software methodologies, specifically scrum with a Kanban board, and adds all bugs we find to our Kanban board. Our Kanban board is digital and similar to Trello in many regards and we have a single list for bugs... We end up with duplicates and now have a long list to try and scroll through... Has anyone run into a similar situation or do things differently that work well for their team?
The original submission ends with one idea -- "I'm thinking about pushing for a separate bug tracking system that we pull bugs from during refinement and create Kanban cards for." But is there a better way? Leave your own experiences in the comments. How does your team track and manage bugs in your software?
The original submission ends with one idea -- "I'm thinking about pushing for a separate bug tracking system that we pull bugs from during refinement and create Kanban cards for." But is there a better way? Leave your own experiences in the comments. How does your team track and manage bugs in your software?
Production Trouble Tickets (Score:1)
We count the cases in production. That's how we track bugs. :-)
Re: (Score:2)
Seriously, we used to use bugzilla but we have since integrated the functionality into request tracker.
Re: (Score:2)
...and integrated our time reporting system into it since there is already room planned for it in RT.
Re: (Score:2)
I switched from Bugzilla to Retrospective about 10 years ago and never looked back. Huge improvement.
One of the best features is that it was abandoned around 2010, so it is perfectly stable with no code thrash. Just keeps working the same, year after year, except when I add a plugin for a sort option or something.
I love abandoned code. It has no bugs; only errata.
Re: (Score:3)
>One of the best features is that it was abandoned around 2010, so it is perfectly stable with no code thrash. Just keeps working the same, year after year, except when I add a plugin for a sort option or something.
I love abandoned code. It has no bugs; only errata.
Finally, someone who feels the same way I do. I like to find an application that isn't constantly being updated needlessly and is "abandoned". I build my own stuff into it, improve it as I see fit, harden it so it's secure, and never worry about some craptastic new release breaking shit and removing or mangling features I rely on.
And no, I'm not being sarcastic. Like you say, it just keeps working the same, year after year. What's not to like about that?
Re: (Score:2)
And we just close those tickets with a note that says "working as designed". If it's important enough, the customer will escalate it through management and pay for more software customizations.
Bugzilla (Score:5, Interesting)
Re:Bugzilla (Score:5, Interesting)
But all of the other solutions suck worse.
We use JIRA, and have about 45,000 open issues (word Atlassian uses rather than bug). The search sucks, and it's nearly impossible to find what you're looking for. At least the Bugzilla bug list page has some sort of logic even if you can't tell what it is. We do four hour bug grooming meeting each week to prioritize bugs, but that typically gets ignored and customer complaints are always what is pushed to the top. Our workflow with JIRA is so overly complicated and depends on who opened the issue, the type of issue, and other options so everyone is always confused as to how to handle workflow even though we've used it for over five years. We even hired an expensive Atlassian consultant to help, but $8k later he simply made things more complicated. Bugzilla is pretty bad, but at least you know how to use it.
Re: (Score:3)
Re: (Score:3)
It.s been a long time, but I really had some experience with bug tracking starting back when the earth was young and the IBM Model 026 cardpunch ruled all. And thereafter for a number of decades.
Some thoughts:
Re: (Score:2)
If you don't fix your bugs, then technical debt will begin to pile up and overwhelm you. In fact, one of the quickest ways to judge the quality of a codebase is to look at the number of bugs in the bug tracker (of course that can be gamed also, like any metric).
Re: (Score:2)
Oh mother fucker, if you have 45,000 open bugs that's clearly wrong.
It's so far past "wrong" they need to invent a new word for it.
If your application has 45,000 open bugs I'd have a hard time understanding how it could even run or stay open long enough to click a button.
Re: Bugzilla (Score:2, Interesting)
Same experience here with Jira! Even after three full days of training when a group of us was hired last fall, we still usually don't know what to do. The having different workflow based upon hidden fields is just confusing as hell.
Re: (Score:2)
I've used JIRA with 2/3 (second company was acquired) companies, and it worked well for us in all of them. I guess if you have competent administrators, you're good to go. Blaming the tool is the refuge of the incompetent.
Re: (Score:2)
We use JIRA, and have about 45,000 open issues (word Atlassian uses rather than bug).
You use JIRA, so 45,001. :-)
Re: (Score:2)
We use RM Track... The biggest issue is the way bugs are assigned to people and everything is controlled with strict user permissions. Bugs get stuck with people, others can't even comment or take them over or contribute to them, the engineers can't mark stuff as "not a bug" etc.
Even with fairly small teams, you want something open like Github has. Too many systems try to micro-manage the process.
Re: (Score:2)
Do you seriously have 45,000 open issues? I'd say your choice of bug tracker is the least of your problems... How on Earth did you get into a situation that bad? Do you expect to ever get out of it?
Anyway, my two cents: we switched from Bugzilla to Jira two years ago. Bugzilla was ugly and painful, but it worked. Jira is beautiful, but I can never find anything. To be honest, I preferred Bugzilla.
Re: (Score:2)
I work for a Fortune 50 company and that 100% sums up my experience as well. Bugzilla may be "dated" but every else is even crappier. At one game job I've used TestTrack [wikipedia.org]. Bugzilla just works.
EVERYTHING in Jira is slow and OVER-ENGINEERED. i.e. In Jira how many times have you accidentally created a "story" instead of a "bug"? Uploading screenshots is still a PITA. The entire workflow -- holy crap -- how do you screw this up this bad???
But management has to keep trying to justify sniffing the Atlassian gl
Re: (Score:2)
We use JIRA, and have about 45,000 open issues
Your problem isn't the bug-tracker. Switching to a different project management tool will not help you.
It will help if you ditch your bug-prioritizing meeting and spend four hours a week each picking bugs and random and fixing them. In fact, make it an eight hour event.
We even hired an expensive Atlassian consultant to help, but $8k later he simply made things more complicated.
Yeah, you should have hired someone to fix bugs. That would have actually made things better.
Re: (Score:2)
45,000 open bugs?
What kind of shit-for-brains programmers are you clowns hiring? Please tell me the name of your product so I can make never to come within 1000 meters of it.
Re: (Score:2)
I would take JIRA over TFS any day of the week.
Re: (Score:2)
Bugzilla. It sucks. But that is what it is there for.
Same here. It sucks but does the job (and it is better in many aspects to the alternatives.) So bugzilla for product bugs (that directly affect a product's ability to meet requirements.)
Anything else goes into git issues.
Re: (Score:2)
Bugzilla. It sucks. But that is what it is there for.
Same here. It sucks but does the job (and it is better in many aspects to the alternatives.) So bugzilla for product bugs (that directly affect a product's ability to meet requirements.)
Anything else goes into git issues.
Oh, and we triage the shit out of both (bugzilla bug entries and git issues) as we go along as well as once a month before the start of the next sprint planning. In fact, a mandatory all-team scrub is scheduled at the end of each sprint. Each developer is expected to look at the standing list regularly, just briefly, nothing epic, and make a list of things that need scrubbing.
No scrubbing == insanity.
IBM Jazz Requirements Server (Score:2)
Or some such nonsense.
JIRAA (Score:3, Informative)
Re:JIRAA (Score:5, Funny)
JIRA, plus great integration into 'fr agile' methodology.
We use JIRA as well, but fire all US development employees shortly after each acquisition, in order to replace them with cheaper developers in India. The developers in India do not have access to JIRA, and the few that do have access generally ignore it. Ever 4-5 years we'll delete a project queue to clean up the tickets once our customers choose not to renew their contract.
Is this not how everyone else does it?
Re: (Score:2)
Sounds very efficient. Have you thought about automating project queue deletion?How about transferring your development from India to Swaziland? I believe that may be the coming thing.
No bug tracking (Score:2)
We begin fixing any bug as soon as it is discovered.
Re: (Score:2)
Re: (Score:2)
That's what we do.
We track in problems in email and sometimes also on a separate website for bug fixes / enhancements we want to expose to customers.
We get to fixing as soon as a bug / inconsistency is discovered (usually by us and not the customer). It's chaotic when big changes are needed, but because we only have 2 or 3 people working on it at any given time it mostly works and allows us to respond within minutes or hours, not in 2 week "sprints".
Re: (Score:2)
Re: (Score:2)
We begin fixing any bug as soon as it is discovered.
An excellent practice, if you can do it without disrupting your workflow or destabilizing your codebase too much.
But even in the best-case scenario, where every discovered bug is correctly fixed within 5 minutes of you learning about its existence, have a bug-tracking database is still a big help. That way when another user reports the same bug a few days/weeks/months later, you can go back to the bug database and refresh your memory about what the bug was, when it was fixed, and what versions of your soft
Re: (Score:2)
Re: (Score:2)
(I'm sure this will be frowned apon) but we're an online advertising network that's been around since 2007. Maybe we're just really good at not introducing bugs. Or luckier (from your description) that we're not using microsofts toolchain. We also don't do much of what you'd call meetings either.
Re: (Score:3)
4 hours of meetings every day? You have 4 half-time devs, and you're wondering why you can't keep up?
Re: (Score:2)
If I had four hours of meetings a day, I'd find another job. I already mentally mark the end of the last meeting of the day as time to get serious work in.
Re: (Score:2)
So, from someone else who has been in the same trenches:
Most organizations/teams are absolutely terrified of losing information. "What if this is needed/important someday??" What they don't realize is that first, that information is usually never used again; second, it can usually be found again pretty easily; and third, that information quickly becomes outdated with respect to the working code and is consequently MISinformation, not actual information.Then there is the cost of dragging it around, meetings
Redmine (Score:3)
We use Redmine with an assortment of plugins.
(Recurring Tasks, Base Deface, Base Rspec, Customise Core Fields, DMSF, Edit Custom Fields, Git Remote, Wiki Extensions, Redrisk, Scrum)
Works great for us by integrating development and maintenance. + it's free and sits on its own little linux VM.
Re: (Score:2)
Same.
For internal projects or where the customer doesn't specify, we use Redmine. It's simple and easy and with a few tweaks you get it working the way you want it (it's open source ruby on rails).
The Wiki is good for documenting stuff like setting up the build environment and all that, and the issues page documents all the issues with custom fields and all that. It's simple, and searches are not like Bugzilla with a million fields, and it's easy to get to a "all bugs" list so you can reverse sort (highest
Re: (Score:2)
I work at a small company with 4 IT staff. Redmine suits us fine. Open source, locally hosted and the application users soon worked out how to submit and track issues.
We used to use Mantis but wanted some basic project management features. The subversion repository integration is nice too.
Don't write buggy code (Score:5, Funny)
My team gets around this whole issue by simply writing bug-free code from the get-go, and moving on. Saves a lot of hassle!
Re: (Score:3)
Re: (Score:3)
Have you managed to ship Hello, World yet?
No, it didn't compile.
We rely on unhappy customers (Score:1)
To track our bugs. No need for duplication of effort.
Re: (Score:2)
To track our bugs. No need for duplication of effort.
Simply call them "features" and charge extra for "upgrades" - problems solved.
Use a bug zapper! (Score:3)
We implemented a rule where whenever a bug is found in code, the coder has to go lick the bug zapper. It only took three hours... before we killed all the developers. It didn't fix our existing bugs but we haven't had any new bugs (or code) since! ;)
simple is better (Score:2)
We invert the problem (Score:5, Funny)
Simple (Score:1)
Blame it on the other team.
Issue tracker ... (Score:2)
Depending on organization your let someone confirm the bug and avoid dups, then you put it on the Kanban board. Alternatively you have on inside of the team confirming it.
But: why are you mixing Kanban with Scrum in the same team? That does not really make sense. Unless you switch from sprint to sprint to one or the other paradigm.
More or a process problem than a tool problem (Score:1)
I don't think the problem here is necessarily the tools, but might be the process. Do you ever groom the backlog/bug list? How do you prioritise your bugs? You may wish to look a bit on the team culture and think carefully about how you are dealing with older tickets.
It also sounds like you are building up a debt of bugs, which isn't necessarily the end of the world, but you may want to ensure tha
Re: (Score:2)
I don't think the problem here is necessarily the tools, but might be the process
Bingo, bug tracking is hard, you need a dedicated admin resource who understands the software to manage the open docket list for any non-trivial project. It the same as source control, you don't let everyone check in whatever the hell they want and stay in business. Also a customer facing tool is not a bug tracker, it's a customer complaint tracker.
Re: (Score:2)
There's no point in tracking bugs for long. If you didn't fix them withing a couple sprints after they were reported, you never will, so you might as well auto-delete them. Nothing of value will be lost, and suddenly you can easily see all your bugs and pick the high-priority ones.
Re: (Score:2)
I don't get how Scrum even works. The last couple places I worked had daily standup meetings which were at least 1-2 hours, and in one company's case, 4-6 hours. Every. Single. Workday. Plus, the meetings didn't do anything other than just get the same lazy trolls to keep stirring the pot. The PM would go around and ask what people did, and the same people would be wailing and saying that they were blocked by so-and-so, so they cannot do any work today. The person who was supposedly blocking then had
Re: (Score:2)
The way Scrum works is, among other things, to have fifteen-minute standups that are focused. One way Scrum fails to work is to have standups that last over an hour and accomplish nothing. That will also discourage the developers, and the particularly good ones will go to a company that doesn't use buzzwords to excuse interminable meetings and constant harassment.
Re: (Score:2)
Re: (Score:2)
Triage (Score:3)
OP needs to triage the list he has, not duplicate all his bug tracking problems. One pass to get the list cleaned up now, then implement an ongoing practice of reviewing new submissions.
Bug tracking? Let me laugh harder. (Score:1)
I'm in operations, not development, so I only do some little scripting, but I sit right by people who are apparently honest-to-goodness software developers. This year, I've at least gotten them to accept some tiny bit of structure and (in theory) the idea of testing things somewhere other than the production systems.
Most of our code was written by brainiacs who have since left. A lot of stuff is just a "black box" to almost everyone. We rarely know how or why it works. When it stops working, we rarely k
Re: (Score:1)
If the team can't figure out the current implementation, having them re-architect it will almost certainly be a disaster that will make your current problems look like nirvana.
First, deprogram from the Agile cult (Score:4, Insightful)
By which I mean, this isn't a collaborative process, or more precisely, it's not improved by collaboration, as so many things *aren't.*
Then, use any database that makes sense. Preferably one hosted on your local server so it's not deathly slow and/or intermittent. Tack on any UI with a minimum of interface fluff. Add keywords. Using keywords, scan periodically for duplicates. This needs to be ONE person's job, preferably done for the first hour of each day. Never, ever fire that person.
It's a human language problem. AI isn't good enough. You still need a person familiar with your code and package to review it.
It's expensive. This is software. You have to pay to play.
Keep the numbers low (Score:5, Insightful)
If you do not, no amount of "magic" management and tracking will help. If you keep them low, however, you do not need a lot of tracking, every bug will be unique enough to be memorable or fixed fast enough to not need tracking.
Of course that means you need to have really good architects, designers and coders. Hard to get but worth the price they will ask.
Re: (Score:2)
If you do not, no amount of "magic" management and tracking will help. If you keep them low, however, you do not need a lot of tracking, every bug will be unique enough to be memorable or fixed fast enough to not need tracking. Of course that means you need to have really good architects, designers and coders. Hard to get but worth the price they will ask.
As well as clear requirements, plenty manpower, no deadlines and a free pony? In any case, straight bug fixes aren't usually the problem it's more vague issues that either aren't important enough or too rooted in the current design to just fix but ideally you'd gather up, make coherent and implement the next time you redo an interface, module or something like that. On the one side we don't want to simply reject it and forget about it, on the other it's no good if it essentially goes in the garbage bin beca
Re: (Score:2)
Re: (Score:2)
Only if you personnel is mediocre. Which is the standard-situation, admittedly. That the result is not very good is also standard.
osticket (Score:1)
OSTicket http://osticket.com
Trac (Score:5, Informative)
Re: (Score:2)
LOL, fair enough!
I missed that typo, I meant to say Jira, which costs thousands of dollars and does basically the same thing.
SCRUM/Agile - the problem (Score:1)
SCRUM and Agile had a great idea - improve the process - but often the implementation of these makes it worse. Just like religions in general .
even more agile (Score:5, Interesting)
I wait until some manager comes by my desk and asks me to stop whatever I'm working on and fix whatever bug someone just complained about.
Re: (Score:3)
Re: (Score:2)
Sounds like you have a cheap and functional prioritization algorithm. Perhaps you should consider selling your managers?
I'll sell you mine, in fact there on special offer right now!
Easy... (Score:1)
We just open tickets describing the issue and toss it over the fence to the team in India to handle.... Why would we want to waste our time fixing bugs when we can pay somebody else to do it? ;-)
The agile way and the real way (Score:2)
The agile way is to keep it as a backlog item until the PO feels its time to fix it.
The real way is to listen to what front line support say and fix it when the noise becomes too much.
If its not a priority for anybody, then no one cares about the bug.
Dafuq is this even an issue? (Score:2)
this question really needs to be asked?
Re: (Score:2)
Is there a canonical answer to what you're supposed to do when you're doing scrum/agile/kanban/something involving sprints and you "get your job done"? In other words, once you finish all the work assigned to you for the current sprint, do you start trolling the bug database (what if you org doesn't have one?), tell the project owner you're out of work, pick something from the backlog that looks appealing, play warcraft, or what?
Re: (Score:2)
Ideally, you pick up the sprint task from your team-mate who is falling behind, so that the team as a whole meets every deadline. I worked on a team that actually worked that way once - it was pretty amazing. Only time I've ever seen scrum done by the book (e.g., managers banned from daily standups).
Re: (Score:2)
?), tell the project owner you're out of work, pick something from the backlog that looks appealing, play warcraft, or what?
https://www.youtube.com/watch?... [youtube.com]
Re: (Score:2)
Nothing to stop something else being pulled from the product backlog into the current spring backlog. When you know your velocity in terms of the amount of story points you can tackle in a fixed period you should know what size story can be pulled forward for you to do in the remaining time for the sprint and still result in completion for that particular sprint.
Same diag tools for server issues.. (Score:1)
An ouija board and a pack of hunting dogs.
Who cares? (Score:2)
They're all basically the same
RT, Bugzilla, JIRA, Traq
Pick one.
Depends on circumstance (Score:3)
I am more of a half-hardware, half-software person, and was involved in very different projects, each with their own ways of doing it. Here are examples which did work:
- ClearQuest (ugh) with very draconian rulesets. This is for a team that has a long history of doing things, and with at least 100+ people developing something and there always are some people who won't do the right thing. Not much tight synchronization between contributors required.
- One gigantic e-mail thread where one person summarizes open issues every week, and closes them when the right code shows up on the master branch of the git repository. This involved roughly 4 people writing the code, writing test suites, writing build systems, and dealing with customer crisis. We agreed that any overhead to setup/maintain a proper bug tracking system wasn't worth the effort given the expertise of the people involved (Verilog hardware designers). Need quite a bit of tight face-to-face synchronization, but that was fine for everybody.
- Simple Bugzilla.
- JIRA with very lax policies (anybody can create, reassign, close tickets. Basically no admin/manager at all)
- Gigantic Excel table maintained by a manager/moderator.
What I learned was that
- A tool is just a tool - what actually matters is the people. Claiming a human problem to be a methodology/workflow problem isn't going to make anything work.
- Start simple - any workflow that tries to anticipate all the possible things that can go wrong will end up being overbloated, and the methodology itself will be ignored because (again) the people doesn't wholeheartedly agree with it. Start with something bare minimal and build up as you (and everybody else) understand the culture of the team.
In this case the original submitter seems to be concerned of seeing duplicates... I'd recommend one person volunteering to be the 'duplicate cleanup' agent for a while, and let that person try doing it manually for a week or two, and observe where/how dupes get created and what the cleanup job consists of. Or, just live with the duplicates and close them as they appear, if the amount of duplicates isn't that much.
Focus on human processes (Score:2)
Lots of people have responded suggesting various tools or whatnot. It is quite likely that you already have a good (perhaps not the best) tool for the job, and what you actually need to do is change the processes that humans are following.
The best way to approach this is to use the "5 whys" approach, where you state the problem (or group of problems) and ask "why" 5 times to try and work out the root causes, then address them.
Eg, Problem: We have too many duplicate error reports. Why? Because testers don't
CMM level 1 (Score:2)
I've had a meandering career which went from academia to industry programmer and back to academia. My new academic field (phylogenetics) can have a fairly high programming content, but universities don't usually have the level of organization around programming that an industrial organization does.
Shortly after I started my new job, my boss/head of lab came to me. He'd received a survey asking organizations about their programming procedures (i.e. much like this question.) Having learned about CMM (Capabili [wikipedia.org]
Re: (Score:2)
P.S. actually answering the question, from the point of view of the company I used to work for (15 years ago).
When I started, we used a home-grown job/bug tracker. We transitioned to ClearCase (source control) and ClearQuest (job tracking). There was supposed to be wonderful interoperability between them, but it turned out that (at the time) this interoperability only worked in Windows, and we were a Unix shop, so I (and some others) had to cobble together some interoperability of our own. I wrote a daemon
Avoid monolithic "components" (Score:2)
At our organisation I'm responsible for two large (sub)systems -1 DB with more than 6E12 records and one critical online and batch document processing system- and a handful of smaller ones. We don't run any software to track bugs.
We work in a way to avoid the chores of tracking bugs by dividing our software into smaller components with clearly defined interfaces. Typically, after a release where testing wasn't what it should have been, an overseeable period of instability starts and issues are dealt with
Newspapers (Score:2)
Basically when you buy a dead tree newspaper the severity by which any bug should be addressed is related to which page it appears on. If your software is mentioned on page 15 then you're good. If you hit the front page then you have a critical vulnerability and you should move to addressing the bug.
Post its, Excel sheets and E-Mails (Score:2)
We're an agency.
I'm leaving at the end of this month. :-)
Simple... (Score:2)
Hey, you. The customer complained about a bug, go contact him and solve it ASAP.
First you have to decide what type of defect it is (Score:2)
What we do is first classify variations of bugs. During development testing cycles, any defect found in testing becomes a defect subtask, and is prioritized right along with any other work subtasks under our current feature.
After code is deployed into production, any reported defects are captured as a higher level defect ticket (not subtask) against the application/component, not necessarily a specific project effort. If the defect is a true production issue that must be corrected, necessary work subtasks
FogBugz (Score:2)
It works fine, it's easy to understand, it doesn't get in the way, and it more-or-less does what we want, so we can spend our time wrestling with the bugs themselves, rather than fighting the bug-tracking system. We've used it for about 10 years now, and we're happy with it.
Tools vs. Processes (Score:2)
Stick with whatever tool your team knows how to use well... switching tools too often isn't going to make up for the minor efficiency boosts for upgrading your ticketing system. That said, maybe you want to use a new project / product / major release to use as an opportunity to introduce (but not test -- you've already done testing using a throwaway project, hopefully) a new ticketing system.
Kanban has a triage process, where you spend a bit of time doing some housekeeping on your tickets to find and elimi
Tracking bugs isn't that hard... (Score:2)
All you need is a decent tool and a reasonable methodology, and of course someone to track and enforce it.
I've used everything from a home-grown system to JIRA to Quality Center to TFS. I've seen basically no enforceable rules (which sucks) to ultra-precise rigid workflows (which really sucks).
We once had a severity 1 issue come up which we quickly fixed. Turned out it wasn't a sev1 by definition (no workaround) but since we fixed it quickly we didn't care all that much. Then a senior manager saw we had
Re: (Score:2, Insightful)
Re: (Score:2)
What ad revenue? I use adblock you fat retard.
You're the exception then. Everyone else doesn't use adblock.
Re: (Score:1)
the slashdot comments however do generate material for me to discuss on my blog, and non-slashdot people who visit the site do want content to read. it takes me hours and hours for all those books and blog posts and comments, but what else m
Re: (Score:2)
You shared your delusional karma point total on here, care to share your analytics for your website? Sales figures?
Nope. Confidential business data.
Re: (Score:2)
Partly though I think because if the close it and it isn't fixed,
it's their problem rather than the sw devs problem.
If they haven't either closed it or marked it as not fixed, isn't it still their problem? And how do you release with a heap of untested code?
Re: (Score:2)
And how do you release with a heap of untested code?
You just build the installer, and upload the package. Exactly the same with tests, or without.
If it was different, you'd have to account for all those times where you tests, they just didn't test for the bugs.
Re: It's hard, but it's important. (Score:1)
Why? Mantis Bugtracker will do all of that for you plus provides an API for external consumption/modifications.
There is literally no reason for homegrown software anymore. Just search GitHub and you'll find 3 good solutions and 10 solutions that, all integrated together, work just as good as what you have to maintain.
Last company I was at, they shat on Jenkins because it was too slow (50 avg builds using a 2 core 4gb aws VM.. yeah) so they wrote their own. Then complained that it doesn't scale, doesn't ha