Saturday, March 29, 2014

If it ain't broke don't fix it!

Most recent update explained from a slightly technical point of view.

After last week's update some of our users experienced problems with their roles and accessing their students, courses and programs. This lead one of our favourite clients to exclaim the title of this blog posting. Of course we worked quickly to resolve any issues but I wanted to explain to her and the rest of our clients why we made the updates and why we felt they were necessary.

Last week's update had three major components and a bunch of smaller bug fixes and enhancements. I'm only going to discuss the major changes as they had the most impact. The major components were grouping and reorganization of the modules, simplification of roles, and changes to sharing.

The grouping and reorganization was mostly aesthetic and wasn't the cause for any of the problems although we did get some support requests asking where things had gone to! This change is the first step of collapsing all the navigation into a single menu bar with drop downs along the top of the page. Shifting navigation to the top of the page will free up the whole width of the page allowing us to present information more clearly and quickly. For example, with this change we plan to present the entire student record on a single page - no more tabs to switch between profile, events, documents etc...Collapsing the navigation will also allow us to take better advantage of responsive web design. When complete, ampEducator will be independent of the device you're using and all functionality will be available whether you're on smartphone, tablet or desktop computer.

The simplification of the roles ties in directly with the plans mentioned above. When ampEducator was first designed it consisted of navigation going 4 levels deep. For example you might be in the Configuration Section, Role Module, List Role Menu, Add Role Submenu. There were good reasons to do it like this at the time. It gave a clear structure to the application which simplified interface design, security and development. Over time however it became clear that there were problems with such a system the biggest one being that most functionality never required navigation going 4 levels deep - in fact most of what happens in ampEducator could be done with much less. The roles were built around the framework and provided very detailed control over access. You could specify which level a role could access (the section or module or menu) and the type of access (create, read, update, delete and print). With a dozen sections containing a handful modules which in turned contained a bunch of menus and submenus adjusting roles wasn't easy and we wanted it to be easy. Therefore as we reorganized and grouped the sections / modules we simplified the roles giving clients the ability to decide only which sections and modules a user had access to and whether or not the role could read or read / update the information.

One look at the new Roles page can quickly tell you what a user can and can't access.
For most if not all our clients we didn't expect anything to change much - not surprisingly most clients never had a need to adjust role access at it's finest levels. The problems arose when we converted the finer access of the old roles to the simpler coarser access of the new roles. This caused some users to lose access to parts of the application they previously had access to. We were also not comfortable having users spread out over 2 different parts of the applications. Admin users could be found under the Institutions module while other staff were found under the Staff module. We felt that for simplicity and consistency all staff should be under Staff and so created Staff records for the admin users and incorporated them into the Staff module. Tthe Admin User module was removed altogether  When incorporating the staff we also changed their roles from Admin to Location Manager. Technically they are different roles but their access maps are nearly identical with the exception of Location Managers not having access to Institution details by default. We wanted to reserve the Admin role for the Administrator of the Software and our point of contact for the institution.

Lastly we changed the structure of our sharing tables. The role access above gets a user read or read / update access to a module of the application. In that module, however, there might be objects which they should not be able to see or edit. For example all Recruiters should have read / update access to the Prospective Students module but they should only be able to see and update students assigned to them. This assignment was previously done through a single Shares table. A row in the shares table consisted of identifying the share type and id - i.e. Student #100 - whether or not it was related to any other shares - i.e. Document #30 related to Student #100 and who had access to it, a user, a group, if they were the owner of the share and whether or not they had update access. When a recruiter lists their students the application searches through the shares table to find all prospects shared with them and then fetches those prospects. Since this sharing occurs at multiple points of the software (Courses, Documents, Emails, Events, Letter Templates,  Prospects, Reports, Students) the Shares table plays a significant part in the performance of the application.

Several years ago everything seemed fine with the way we had designed the sharing table. However as use of the application increased we noticed an alarming trend with the sharing table, it was growing too fast. In some cases the Sharing table made up 20-30% of a client's database. While the database (and more importantly our servers)could handle the current size we knew that as the number of shares climbed into the millions there would be performance problems. Where did all these shares come from?

  1. Each object that needed to be shared was shared automatically with all admin users. We didn't want to write extra / different code to specially treat admin users so we had decided that all shares would automatically be shared with admin users so that they would have access.
  2. For each related share, the shares of what they were associated with were copied. If a student was shared with 1 advisor and 5 admins when you added a document it would also be shared with the 1 advisor and 5 admins.
With 100 student you could see 10000 entries in the shares table. Make it a couple of thousand students and many more prospects, add some courses and the number of shares were up to several hundred thousand. When email support was added the sharing spiked and for some schools the sharing table consisted of millions of records. We needed to get the shares table under control otherwise performance was going to start to degrade and we had to do it now before it really got out of control. We did the following.
  1. Split the shares table into 2 tables. A shares table holding the information for the share including the share type, location and whether or not it's related to another share and a share access table containing references to those users or roles who have access to this share.
  2. Stopped adding shares access for related shares and instead decided to look it up from the share it's related to. When you now add a document to a student record, it is tagged with the student's share. Only users who have access to the student have access to the document and we don't have to add any extra entries for the document!
  3. Treated admin / location manager users as special cases in the code instead of automatically sharing everything with them.
With these changes we reduced the size of the sharing table by 90-95%! This should result in perceivable performance improvements. Most of the issues users experienced were as a result of #3 - treating admins and location managers as special cases forced us to re-write the parts of the application which dealt with security and lists. Unfortunately when you re-write code there is always the risk of introducing errors. We spent weeks debugging these but one or two problems did sneak through causing some users to lose access students, prospects, courses etc.. After some tweaks to the code everyone had access to everything they should and in the end we were relieved shares table is now under control.

We are happy with the updates. The implementation was bumpy but I think we've worked out all the issues. With the roles simplified and sharing under control we can move forward to add new features to ampEducator. Some of what we have planned over the next few weeks / months include.
  • Analytics for all sections
  • Campaigns module to better track and manage your email marketing.
  • Document Storage increases - 50-200 GB per institutions depending on your plan
  • SMS Alerts - 1000-4000 SMS messages per month included in your monthly fees
  • Accounting improvements including a proper ledger, automated invoicing and statements
  • Finacial Aid module
  • Housing module
As always we welcome all feedback and ideas for improvements. Please email us