Video Transcript
Presenters:
Chris Bland | CEO & Co-founder | BDQ
Dom Bush | CTO & Co-founder | BDQ
Chris Bland: Hi. My name's Chris Bland. I'm a certified Atlassian professional. I'm one of the owners of BDQ which is a solution partner based in the UK. Dom?
Dom Bush: I'm Dom Bush I’m also an Atlassian Certified professional and also co-founder of BDQ.
CB: So then in this video, we're going to talk in a little bit more detail about some of the migration tooling we've developed and why we've done so, basically. So this is probably a follow on to our other video where we went into a high level around how the migration process works. This time we're going to a little bit more detail and it will be a shorter video. So what would you say some of the more thorny problems that we've dealt with?
DB: Well, one of the really difficult ones is that when you migrate to cloud, the users must be unique in the sense that they must have a unique email address. And it's quite common that that is not the case in most users, most customers databases. So it’s unique-ing the email addresses or - the other issue about is some users in user directories don't even have an email address, but unfortunately, on cloud that is one of the key identifiers for a user. So you have to make sure that that information is available.
CB: And that has various knock on effects doesn’t it about - because actually users own things, they have access to things. So it's not just making them unique, it's making sure you've got the correct unique list and then everything is transferred correctly, basically, and then those people have the right access in terms of permissions.
DB: And it could be the case that also you have inactive or deleted users that own particular parts of the configuration of the products in which case those things will not make it over into cloud using the Jira Cloud Migration Assistant or the Confluence Cloud Migration Assistant, it's really important to get this particular aspect correct.
CB: Yeah, no absolutely, otherwise it causes no end of trouble doesn't it. And one of the other things is, you know, in terms of when scale increases and we have larger, older instances, I think, yes, you get a lot more users, you know, potentially thousands of users. As you said, there can be multiple use of directories involved, a lot of duplication.
All the issues you talked about, such as permissions not being right or inactive users owning things. Which, then won’t migrate over with standard tooling. There's a bunch of problems. There's, there's a series of other problems as well, isn't there? Such as dashboards, filters, custom fields, automations and so forth. The bigger the scale, the more of them there are. And JCMA, I actually, basically, doesn't deal with some of those things out of the box at least.
DB: Absolutely. So for instance, the default functionality in JCMA is that filters, boards, dashboards do not migrate. There is a dark feature which is sort of like a preview of functionality that's coming to the tool later that does move some of that information over. But yeah, there are issues around these things. And it really, as part of the migration process that we go through is understanding all these different aspects and then creating a runbook for the actual production migration over a period of a number of different iterations. In order to address those issues, make sure that by the time you get to cloud, you have everything that you need to support your business requirements.
CB: Yeah, no, exactly. And actually, based on some of our development experience and the way we prefer develop, we've actually gone for our sort of a test first approach, where possible, to try and make this more scalable, haven’t we. Some of the tooling you’ve developed and other peolple have developed as well. Yeah, basically tries to make sure we can do as many preflight checks as possible and make sure with a known, known case at all times.
DB: Yeah, absolutely. So Atlassian provide a number of different tests, which they call the preflight, which are pieces of SQL that you can execute against the underlying repository under JIRA to identify some of these issues that we have taken that that outline, if you like, and added additional tests for various other different problems that we found during previous customer migrations.
What we've done on top of that is a harness that runs these tests in a sort of unit test kind of way that gives us an output in a report which we can look at and say, yes, we're seeing these particular problems. That’s very useful, be able to just rerun that over and over and over again while we're developing this runbook, it allows us to sort of use those software development kind of approaches, and we could really see those the number of problems decreasing, decreasing, decreasing until we get to a point where we feel that we've identified all those issues, resolved all those issues, and that we can be confident that when we move into production migration, that we're going to get the result that we all want.
CB: Yeah, yeah. No, exactly. It saves an enormous amount of time, doesn't it, basically and a much more reliable result, basically?
DB: Absolutely, definitely.
CB: So if if you're going to do it yourself, definitely try and do it in a test first type of way, understand exactly what it should like - should look like. And then actually you can take a sort of an approach where you do the migrations and if there's problems, you'll be alerted to them as early as possible and you can correct them as quickly as possible. Whist keeping the situation known at all times. The other thing is we also have a scalable approach to UAT as well, because some of these larger instances have thousands of users involved and they’ve all got their own workflows, projects, boards and so on and so forth. So during that UAT process, we need to make sure that everyone has had a chance, or that everone that’s relevant, has a chance to look at things and then we need to be able to triage them in a manageable way.
So that's one of the other parts of our approach as well. So again, I suggest just not using email for that because that can be very difficult to get on top of.
DB: Yes, and there's always a possibility of missing things if you use email. So a more structured approach for capturing those comments coming back from UAT. Moving through the process of addressing those issues is really important because it just really helps make sure that the quality of what you end up with in cloud is the highest it can be.
CB: Yeah, absolutely. Now we've talked elsewhere about the problems of the XML migration method wasn't, you know, had some problems. JMA or JMCA rather is easier to use isn't it. And does what it does better, but it doesn't do filters, boards and dashboards without using the dark features.
DB: Absolutely. Yeah. So the tooling is very good and it's improving all the time. There - there are certain things that you have to be aware of as you go through the process and address those things.
CB: Yeah. And actually one of the things is if you chop things out the instance, so you want to descope the amount projects you've got that can cause problems, can't it? Basically as we know.
DB: Yeah. So some customers take it, take this migration to cloud as an opportunity to potentially take out stuff that's no longer required. So that might be projects that get deleted, that are archived or that kind of thing. The issue comes with things like filters that rely on those particular projects being available. And then on top of that, obviously you've got boards, dashboards.
DB: So by chopping out bits of data, potentially you affect not just that it may have knock on consequences. So some of the tooling that we've created helps with that process, helps with modifying the JQL, for example, for filters to take out missing content.
CB: Well I suppose, you can sort of consider that, you know, JIRA, if you like, is a large database by which you can query JQL. So if we then start chopping bits out of the database, AKA removing projects, then JQL, that was querying those projects, is going to break basically. And that's what's in your filters.
DB: Yeah, absolutely.
CB: And I mean, the other you know, there's two ways of doing the other way is just take the whole thing through and then prove it on the cloud. But still, if you start deleting tons of projects, and there are various tools to try to look at dependencies, but if you do that, then things are still going to stop working. There’s no, sort of easy way is there?
DB: Regrettably, some of these things don't have very easy solutions unfortunately.
CB: But we have done, we've done some tooling around to try and assist with that and we basically which we - which we'll talk about a minute, which is your JQL - JQL Runner and Fixer. Then there's some other issues as well, isn't there? So we want to obviously try - we take a test first approach one to try and minimize the amount of fixes and retries. That's what everyone wants to do during Migrations. Some people may know that BDQ stands for business data quality. We used to do data quality tooling for data migrations, so we're very familiar with the problem of data migrations. In terms of, well, what approach would you say we've sort of done to try and minimize the amount of fixes and retries?
DB: We wrote a number of different reports that pull the information from the source system and the target system, and allows us to do different supports that say over here, we've got a certain number of filters, over here we've got a certian number of filters -where is the difference is between those two things. And that's quite useful when you're looking at things like permissions, that sort of thing.
DB: So yeah, understanding what you have, what you end up with in, in, in cloud is really very useful.
CB: Yeah. It goes a long way before just how many issues have gone over doesn't it. You're basically - the tooling now that we've developed basically lets you check all the actual key parts of Jira, such as you have filters, dashboards and so on and so forth, to make sure it's actually gone across. Because as we know, sometimes these things don't go across and they don't go across silently either. So, you know, basically there's also the JQL Fixer, User Fixer because we know duplicate users is a no no, which, as we said, causes no end to trouble. You've done a -
DB: Additionally we've written some reports which take the log output that comes out of JCMA and CCMA - actually just JCMA - and compiles it into a slightly more digestible format and that that is very useful because the logs are the logs - but an actual report is slightly more readable than that.
CB: Yeah. And that's makes, it makes the whole thing a bit easier. There's also - yes - script to monitor migration progress which again is occurring over time. Yeah. Yeah. Just a query as the rest API does not require - that’s actually quite handy thing for people to do.
DB: Yeah, the user interface within JCMA is pretty good. However, one of the things that you - I think it lacks really is a progress indication saying how far a particular plan has got. So yes, we've written a utility which simply polls the rest API on and gives us a number which shows us the exact percentage that we are the way through and that that really is very useful when you're actually doing the migrations.
DB: Yeah.
CB: The other thing is quite useful is that when you need to make a source system of Jira so let's say we're going to cloud and then they’ve cut all - we want to keep the old Jira system up, we just want to make it read only, but also we might want to back that out easily -
DB: Absolutely
CB: - you’ve done basically a bulk archive permissions scheme editor.
DB: Yes, absolutely. So we have a script which will go through, look at the permissions schemes that are currently in place and it will edit those to take out write access for users. And, as Chris was saying, we can out very rapidly - takes a few minutes at most - to allow access back to the projects in the event that we need to roll back from our migration approach, migration effort basically. So, very useful.
CB: Because there are some situations where you can't just literally have one read only scheme and just apply it to everything apply it back out.
DB: No, no, no. That would be fine if everyone had the same access rights to all projects. However, in most circumstance, customers don't have that kind of very flat permissions approach. So taking what they currently have and just taking out all the “write access” is a is a much more a safe way to do it in the sense that you don't really want to potentially give someone read only access to something that they didn't have read and write access to before, if you see what I mean.
CB: Now, I know you can't really share the comparison reports because obviously they're customer oriented. But actually, and yes, I know we can also test JQL in bulk, which is very, very helpful, isn’t it? We could just pull it out, test all the statements, which basically is a way of telling us where the filters are going to actually work properly.
CB: It’s part of our “test first” strategy. But I think you've got a small tool, haven’t you, which is just a quick example of of actually how you can just throw JQL at a system.
DB: So what I have on screen here is a very simple python utility. I want to put it in a piece of JQL and then just run it. And what it's doing here is taking that JQL executing against the the target system. So you can see here down the bottom, it tells us immediately where there are problems in this particular piece of JQL. We can see that these two projects, ABB and ABA don't exist. So very straightforwardly, I can go to this particular section, rerun it. Again, it highlights where the problem is against the ABA projects and I can delete that again until I get to a point where the JQL has been corrected. Now this is obviously a very straightforward piece of JQL but more complicated things, it can also do this similar kind of approach. So we could take a list of filters from the source system, run it through this tool, correct them. And the output really is to get to a SQL script that we can execute against the source repository to update these filters and make them good and fit for the for the target - target cloud environment.
CB: So, so basically if we're finding that actually our data is being cut out and we've got a whole bunch of filters that don't work after we've established the don’t work, this is just a quick way - isn't it - basically just trialing those SQL - those JQLs, if you like, and correcting.
So, we thought we’d go in to a little bit more detail about how things work, show an example of some of the tooling that we've had to do to give - so that we can have a test first approach so we can do reporting to compare all aspects of on-prem and on server instances, which basically uses a mixture. I think - doesn’t it, of SQL and API calls, go through all the different aspects - obliviously Cloud is API only - and have look.
I hope you found some of that useful. If you've got a simple instance - it will probably be fine to migrate, something more complex - it might be worth speaking to a partner such as ourselves or others, to see if it's worth getting some help to do it. Because some of the more complex systems can be quite, quite involved to migrate over in a seamless way, which does not impact on your I.T. departments and their time.
Anyway, many thanks for your time. I hope you enjoyed that and see you in another video.
DB: Thank you.
Easy Email Attachments
BDQ Migration Analyst
Attachment Manager
Real Signature
Real Signature
