I'm the sole developer of a piece of software that solves a niche problem. People like it, and it's continually being used by old and new users alike.
Every time I release an update, I get complaints about various new features not working as expected, which is normal.
I try my best to address these issues with software updates, but since the software is growing more and more complex, this approach is getting less and less successful.
My way to deal with software updates is to listen to as many reports as possible, figure out common problems and try to address the biggest problem with a successive update.
However, I'm starting to realize that people only reach out and complain if they're actually having a problem (why send an error report if the software is working correctly?) Because the user-base is growing bigger, the chance that a person is experiencing a one-off problem -- a problem specific to their setup, environment or handling -- increases, while the chance that theirs is a common problem decreases.
So I know that _some_ users are experiencing a bug, but I don't know if it's a common bug, a rare bug, a design error or if the feature is even working for anyone at all!
I'm unsure about what to do here. The error reports I get are vague because of my closed-source "if you have a problem, it is the developer's problem" business model/mentality, client-side log files up to this point haven't been detailed enough, although I am working hard to improve that part a lot.
Is collecting anonymized user-data the answer to a problem like this?
TL;DR As my software grows more complex and the user-base increases, I start having more uncertainties about dealing with updates, stability, vague bug reports and "fixes".
Have any of you had similar problems? Do you know of any good resources dealing with software stability? Or does this sound like a clear indication of <insert common mistake>?
I once built a logging system into the app that the user could turn on/off if they were having issues and it would at least record what they were doing at the time and they could send it to me for review. It captured usage and exception info so I could narrow it down pretty quickly but getting them to use it was impossible unless they called and I asked them to do it. I'm not sure I'd like to be bombarded with user data from all my users though.
I now think a better system would be to build in a bug/feature reporting tool and put it front and centre on the menu or toolbar etc to encourage them to use it. Create the form with as many default values for fields as you can to make the report as specific as possible with just one area to describe the bug.
There's not much you can do about updates/patches, I just issue an email and flag them as normal or very important which at least gives the user a choice of when to install them.
For new features, I'm working on building in an interpreter for Clojure (I'm using C# as the main app language but using say Lua or building a simple Lisp interpreter isn't too hard). I want these new features to be 'plug-ins' rather than issue new releases all the time. It will also give the end user a chance to automate what they need specifically if they are inclined and able, saves me some work for little reward :) It also keeps the main 'engine' code smaller and maintainable, once you have the engine a majority of features are just automation anyway.