Accessibility at Scale

[amazonv]

Accessibility at Scale

Type: Breakout

Track: Development

This joint talk from USAA and Workday will discuss how each organization built accessibility automation into their existing testing processes. In this talk, the presenters will discuss:

 

The drivers behind building accessibility automation into a testing pipeline

How to receive executive buy-in for this process change

How implementation went and the current status of the effort

 

 

I'm Ryan bait man from Deque, I'll be moderating this session titled accessibility at scale. I'm joined by Bryan overtime camp, our principal technical architect at USAA and Tom si Cora, director of accessibility at workday. Before I turn it over, let me address a few housekeeping items. First today's session is being recorded and will be hosted on  demand. If you require live captions, you can access those on the session page just below this video  stream. Lastly, we will save the last 10 minutes of today's session for Q&A but you can post your questions at any time during the session in the Q&A widget.

You can find that Q&A widget alongside chat next to this video stream.

With that said, Bryan and Tom, please take us away.

>> Good morning, everybody. Good morning Tom, how you doing?

>> Hey, Bryan, good morning how you doing.

>> I'm doing well. Thanks, Ryan. So today we are going to talk to you about accessibility at scale as Ryan mentioned and I'm excited to have Tom here from workday. As was mentioned, I'm Bryan oster camp from  USAA. Before we get started, I wanted to give you an example. I'm so thankful that we have got folks from really all around the world here. I'm going to give you an example of something that happened recently. For those of you in Texas you may be more familiar than others. We have that thing called snow down in South Texas which we are not used to. So what happened was basically after we had the snow storm, we had to make some changes where our water filters needed to be changed and worked out. So what I did was I have got this reverse osmosis filter. And if you're not familiar with that, basically the water that comes into our house will have several additional filters to make sure that water is nice and clean and not too hard like it is in the hill country. I'm not a real hands on type person but I decided I can go ahead and fix this. So I did watch a few videos on how to change it. I have got these simple filter things that I just pop them out and put them back in.Esey piecey, no  deal. I said I can do this. I watched some videos. The video said make sure you turn off your water before you do this so that way you don't have issues. I'm like this is easy, you just pull it in, pull it out, no problem. What I did was skipped that step, I wanted to take a shortcut and I decided I'm going to pull out the filter. I pulled it out, not a problem. I was having issues trying to get it pulled back in, it wasn't quite working. And I was like let me just force it, because that's always good, let's take that shortcut. When I pushed it in, apparently it didn't quite seat properly. So instead of it working and me being done, what happened was I had this large stream of water just flowing out to the bottom of my cabinet and I did thankfully have a towel, but that towel got saturated in about three seconds, which was the time it took for me to look at it and realize what's going on. The lesson I learned from that was be very careful when you try to take shortcuts and try to do something against what was advised. And as we start this  session, what I wanted to start with, when we look at scaling and scaling accessibility, Tom and I are going to talk about a few different things and ideas, but one of the things I learned was you certainly don't want to take a shortcut with scaling accessibility either. Hopefully that will be a good way to think about as we're going through this session some of the items and ideas that we might have for that and some of the things may make sense, some may not make sense in your organization, but those are things to consider as well. Again, thank you so much. good morning to those of you it that it's morning. Good evening. A couple are really early morning birds and hopefully you have your coffee and are ready to dive in. I'm Bryan oster camp, I'm a principal technical architect.

 

I have oversight over the accessibility testing space within USAA. I own all testing in USA,a. I also own the assistive technology -- we have a focus space I'm working with several other architects on trying to improve our employee assistive technology and really moving that forward in that space. Those are the two areas that I am a part of at USAA for accessibility. I have been at USAA for 18 years, nine of those years was being an architect and I have been in the application space where utilizing testing technology and assistive testing technology as well as now owning those tools and being in that space and being the provider of that technology. So I have been able to see the challenges and advantages on the application side as well as the infrastructure side. I'm going to hand it over to Tom to introduce himself.

 

 

>> Great. Thanks, Bryan. Good morning, good afternoon, good evening and good crazy morning because I think I heard someone from Sydney, Australia joining the call today. My name is Tom si Cora, director of accessibility at workday. And I have been at workday for the last two and a half years and in the space of accessibility. In my prior role I was working on accessibility for the last three and a half years. We're excited to be here to talk to all of you today. I want to start -- Bryan, I didn't know he was going to share the story. I have been in Texas when it snowed and been in London when it snowed. I can testify [Away from mic] I don't know how I made it. I have got to ask Bryan, in the COVID era, Zoom backgrounds are like coffee table books. Bryan, what's up with the medals?

>> Before all the COVID stuff hit, I ran Spartan races. For those of you not familiar with Spartan races, essentially those are races where we run, it could be five, 13, 26, 9 miles where we run and we climb holes and go underneath barbed wire and throw spears and have a fun time getting bloody and sweaty. That's what that is. For those of you that are Spartans, ba review, you will know what that means. Every race we run, we get the finisher medals. I think I've run about 15 or so of those. It's a good fun time for those of you who haven't done it before.

>> I knew that about you. It was interesting though because in between earlier this morning and now I actually ran to my mailbox. I'm a return and also a triath lete. Bryan's got me thinking about challenging myself. I dug out this one medal. This is Ohio  stadium's this is a finish on the 50 run where you get to run and finish inside the horseshoe. Ohio native here who knows how to drive in the snow. By the way, there's no adventure racer future because upon I'm not a very good trail runner. Anyway, on to our conversation. Bryan, we are going to go over our agenda to get it started?

>> Absolutely. Today, like we mentioned, I'm from USAA, Tom is from workday and really what we are going to talk about is some high-level themes that we've noticed both in scaling our DevOps accessibility journey. What we are going to do is go over the individual themes and then how our organization handles it. What I'm excited about, because both USAA and workday are sharing their perspectives, they are not necessarily the same. Some things we did that were similar but there's also some things that were different. I'm excited to kind of share those differences and I want you to understand as you're watching this that there's really no one size that fits all. We know each organization is different. What you tried to accomplish and what you see that we might have done, hopefully you'll take some of the tidbits from each one of the learning we've had and maybe share back with the community what you've learned as well. Three of the themes we are going to talk about today is how do we empower our community to be able to help us build that accessibility to scale as well as secondly will be how do you show that view at an enterprise scale and Tom will share his perspective and I'll be sharing mine. Last Tom's going to talk about how do we catch regressions or what are the benefits of catching those regressions.

>> So it's interesting and I look forward to the Q&A period of today's conversation to see what kind of questions we get on this topic. This is more of a prompt to think about where you and your organization might be in the journey for accessibility. Are you starting with one or two people in the corner of your office that have low or no air cover from your organization?

Or have you just gotten back end for this to be a priority in the organization and trying to figure out what to do and how to level up on this?

And when you really think about, depending on the size of your organization, what are models that actually can work at scale. I think a lot of people when they get into this space look at this as a problem to solve as opposed to some cultural changes as to integrate the thinking of accessibility throughout your organization.

So as you kind of think about where you are in the journey, also look at your organization and think about how you want to approach this from the organizational mindset side of things. It's not just about accessibility and design, coding and keying, there's a lot more to do organizational-wise to get your company to a place where this is just something that is done because it's how we get work done.

Next slide. All right. So you know, many of you probably are also in that situation where some people in the development community look at accessibility as something they will fix later or as I call it "fix in the mix." And many of us know it's not like a record producer where you call and they fix the sound of your band before the album goes out. Here's a smattering of different roles that I think are fairly straightforward when most people think about accessibility where you need to teach people to fish in your organization. I want to talk a little bit about my experience of two different operating models that I've worked in because how we've approached accessibility and how we've tried to solve it at scale has been different, more because of how the organization is structured and the size of the organization. So I'll give you a tale of two organizational models. The first one is what I call a federated or decentralized model of an IT  organization. In my background, I work for a large bank here in the United States and just the largest of the technology population was greater than 25,000 people globally working on technology for the bank.

And the landscape was well over 600 applications that presented itself as a dot com and behind dot com was about 600 applications, many of them back end, transported to the would I be. We had obviously a bank and lots of documents and documentation and PDFs pulled from all over the place, various data sources and other inputs outside of the company. Also, third-party apps and third-party apps that are integrated into dot com experience but you're actually porting out to a third-party vendor who is not necessarily the same accessibility experience as when you're inside the firewall or inside the experience of your company's app. So you're kind of looking at that and thinking how do you approach accessibility and drive that into your organization is obviously very complex. If you're looking at centralized models for accessibility, how do you get into those organizations and drive accessibility across such a large space?

 

 

Then there's another model. Think about another side of the house where you have a more centralized approach. I'll give you an example of where I work today at workday is we call it UI metadata platform approach, I call it a super platform. Most of our development is in a centralized environment. We also work in a foundation of reusable objects as well as reusable layouts. So mostly in the foundation of a product is developed with tons of reusable components. So it's a little bit easier to approach, fix it once, fix it everywhere for a lot of things that that are components in our product. In my experience, how we've approached accessibility and driven it across the organizations has been very different because of the way the organizations are structured. If you follow on to that, this story that Bryan and I are going to talk about for the rest of the time this morning is some of the unique things we've done that actually go beyond what I just described as some of the traditional sides of accessibility. I'll start with the workday story on this is where we're looking into some tooling options to help our community and teach them to fish and get them the right tools for the job, we were looking at some automation tools. I had the benefit as we were evaluating the various vendors, we brought in our director of DevOps into the conversation. I don't have a background in DevOps. We brought him in because he runs a lot of our automation. It was interesting, he asked the basic question:  Can we actually put these tools into our continuous integration build pipeline?

And I never considered that as an option.

So as we looked at tools, that was one of the options that we chose as workday to provide people with the right tooling for the right jobs and kind of introduced this into the space so we can actually have an enterprise view and look at accessibility at scale throughout many parts of our development organization.

That's just kind of to tease you a little bit for the conversation about looking at how we've applied some of the DevTools into more integration at scale. I'll turn it back to Bryan.

>> So the next piece we are going to talk about, the first thing is empowered community. So Tom mentioned about several different aspects of a community and not just having one central area or one central theme of knowledge but really how can we empower that. Tom talks first about his perspective and I'll dive into the USAA perspective as well.

>> So kind of spring boarding off of that, the right tool for the right job, there are lots of things that we can do to actually help the various roles in the organization to improve. And we work off the  mantra that the [Away from mic] expectation through our organization that accessibility is everybody's  job. And going back to that, you can't fix it in the mix when you think of everybody on the left-hand side of the equation trying to fix it in either dev or QA. Really you are reinforcing the point that everybody has something to do and there's nothing necessarily special about accessibility. It's just a new skill set to learn for your role. We have redone some of these things and we use the expression carrot first, stick later. No surprise, obviously we have a very robust training curriculum at workday. We've developed it to the point where we do it like a college curriculum where we're now up to 300 level classes with very different roles and it's not just development QA and design. We actually have training for content management for marketing website participants as well. We are going to talk a little bit about tooling, but one of the ways we also provide assistance to our organization is through the use of office hours with our design and development communities, again, to not create the situation where we create a co-dependency in the organization but we're here to help rather than solve the problems for them. One of the things using the DevTools in our automation has helped us immensely and one of the reasons I say that is because one of the challenges we still have at workday is we are still helping our development community where we collaborate with them as they're working on new features and shipping products to telemebed the success criteria from the development and testing into their geostories, for example. We are not to the point where we have taught everybody to fish and they are fully independent and sustainable but obviously have a roadmap to get there. I want to talk about one story aside from the traditional development model side of things and this really goes down to creating and setting an expectation and how that can drive organizational change. So it was 2019, I had just joined workday shortly before that, we were very fortunate to have one of our executives champion accessibility and we made it one of our top level goals for 2019 to improve the accessibility of our product. Who doesn't want that?

You get some amazing air cover and support to really drive change in the organization. And looking at 2019 we had a lot of work to do on the product side of things. We had the heat map and roadmap of where we needed to make our initial investments in accessibility. One thing that actually happened in this that wasn't on our plan or roadmap or heat map, but our marketing team approached us because they knew it was one of our top level goals for the company and asked us what do they need to do. And even though it wasn't on our roadmap, we certainly looked at it as an opportunity to partner with them. Our initial conversations were about what we need to do to fix it. And we continued to really challenge that line of thinking to say let me teach you how to do this because marketing content and making it accessible isn't a special skill; you just need to learn how to do that. So somewhat of a protracted conversation to make sure we were getting to the right operating model with the marketing team. And in the end, we ended up launching a new marketing website late last year and it's fully accessible. I think the important part though out of this is that after we finished all the work, our marketing team is self-sufficient and working on all new content with -- they work with some third-party agencies in support of that and making sure that everything we put out is accessible and they're self-sufficient at doing that. I think when you really look at accessibility at scale and looking at various roles in the organization, you set an expectation that it's your job, if you learn to level up their skills, by then the tools and resources, that's when you form not just the scalable side of things but also sustainable, things that people can repeat time over time. That's a little bit of a workday story. I know Bryan's got a different look on things.

>> Thanks, Tom. Yeah, so the title of this slide is don't succumb to learned helplessness. It's something we started in the last few years using on our side, which learned helplessness essentially the definition is when an individual continually faces a negative uncontrollable situation and then stops trying to change their circumstances even when they have the ability to do so. So in other words, it's something I know how to do and it's been a while since I don't have to do it anymore, I have learned this  is -- I basically choose to not have the ability to do this. The reason that's important in the story at USAA was several years ago we had a whole separate testing organization. All of our testing, not just accessibility, but all of our testing was done by a whole separate organization. So what we had was, unless you were QA or tester, you didn't do much testing, right?

I know a lot of you are like that's scandalous. I agree. That was where we were. We were like the team can be responsible for quality so let's start shift testing to the left, et cetera. Over the years we have been on the journey to move that testing really to within the Scrum teams, for instance, but then we also had specialized testing. So in our space that's accessibility, security, performance, et cetera, that was still done within the center of excellence area. So what happened was the teams were still learning how to be able to be responsible and automated testing, et cetera, within the organization, but now we have this specialized testing organization that also needs to be moved out to the teams. What we have now is we have really all of our teams are responsible for testing and then we're starting to role the accessibility testing over to the team as well.

So what we started seeing from this perspective was the same thing where we need to roll this out to the organization. Tom mentioned a great example. We need to make sure we provide the right tools, right?

  

  

So one of the things we saw was we had some of our leaders in the infrastructure space say hey, let's start transitioning the accessibility testing over to the teams. So as the architect, one of the cautions that I had in the conversation we had was it's really important before we transition the process and the abilities to the teams, we need to make sure we provide the right tools. So it's really important, for instance, to have a DevTools, pipeline tools, DevOps tools that allow teams to be able to check their coding and run accessibility right away or, again, security performance, whatever it might be. You want to empower the teams. So when you start looking at scaling your accessibility across your organization, make sure you provide the right tools. And also make it easy to transition. So a concept that we have at USAA is we want to make sure, we call it the testing freeway. But basically we want to be able to provide you the ability to have support and training for the tools if you have issues as well as a quick start way to where you can very quickly run something within the pine line. So an important thing that we learned and I think that saved us some challenges was let's make sure we empower the teams, train them, help them understand how to utilize it. And by the way, when you're moving from a centralized model to a decentralized model, whenever those tools provide you insights or have findings, it's really important to have a tool that says not only here's the issue, but here's how you fix the issue and this is why this is an issue. In the assistive technology space, this is what challenges someone would have trying to utilize this tool. So it's really important to understand  that. I think the other thing that's important to remember as well, there's going to be some teams that are going to embrace it, they are going to love it, move very quickly and say we are going to do this. But you're going to have some teams that will never be ready. There's always something, some challenges that they want to keep working through before it gets transitioned. Those are the folks where you have to say hey, let's jump in with both feet and do this. Some teams will be ready, some won't be, but eventually you have to step up and make that transition. When you look at scaling, find the teams that are passionate and excited and want to start using the new tools, but you're going to have lagging indicators as well on teams that aren't quite ready and at some point you have got to have them jump in. That's important. And then lastly, really quality is the responsibility of the application team so you don't need a separate organization, you don't need a separate individual, and there's all sorts of different examples of teams that do have a separate QA and there's examples of teams that have them combined. Really any one of those solutions can work, but something to consider is when you're scaling it out, like Tom mentioned, you really want to help everyone understand the value and the importance of it and not just relegate that to a few individuals.

So next we are going to talk about viewing at enterprise scale.

>> All right. So it's interesting we have been talking about looking at scale and just in general how do you have the model that you can actually ramp-up and still have control of what's going on. When we start thinking about this at workday from a tooling sense, kind of reiterate that my background prior to joining workday was in banking. And I think when I was looking at the tools, there was more of a compliance side of things to really have [Away from mic] and controls, if you will, or a back stop, the last possible point in your construction of your product, how do you just have that last quality control?

But it was really interesting when our DevOps person started asking as we were evaluating tools:  Can we actually put this into our CI pipeline?

It was a question we hadn't really considered until he posed the question and in my mind I thought wow, let's really think big about this. Why do I say this?

  

  

Let's look at some of the other ways that you can test for accessibility and talk about the Axe browser plug in. It's obviously a fantastic tool. It's one of the de facto standards in the industry out there. But when we were talking to Deque about this early in the cycle, it was snapshot and you play your chat and it disappears like a puff of smoke, kind of like Mission:  Impossible. And a lot of times results from those end up being screenshots that get put into other project documentation whether it's Jira or other tools. Coming from my background, always wanted to get to having line of sight into what's going on in the testing activity for accessibility and really what are the capabilities available to us today to get that. So as part of that, we decided to go on that bold step to introduce this into and use the enterprise version of Axe DevTools into the workday pipeline. I'll tell you a little bit about the story of that. I'll say I was very cautious and concerned that if you put something in the build pipeline that you could potentially fail your developers build for, if you don't do it right, you will have the barbarians at your gate. I need to merge my code because I have to ship my product. That really goes back to the carrot first, stick later. So we spent the better part of the year and a half in our organization working on carrot side of the training, the tooling, the right tools for the right job, setting the expectation within our community that this is expected of them so that by the time we went to turn this on, we were hopeful it wasn't going to be this controversial, if we were going to do this thing and turn it on and [Away from mic] figure it out. A little bit more of a process of how we got there as well. Just for transparency, most of our automation in our CI pipeline is through web driver IO and cypress. We did spend a little bit of investment at workday to put some wrappers around them so we could drop [Away from mic] workday. We started with very small pilots. One of our development teams locally here in northern California, and it was really interesting and I'll qualify this by saying I did not do the work to put the wrappers around the Axe DevTools, but I'm told it was fairly easy. It actually was a fairly simple for us to do that and the results were pretty amazing. Share with you that after that, we're like okay, we'll take it to some of the development organizations here in northern California. And my Vice President challenged me right on the spot and said take it everywhere, if it's that straightforward, implement it everywhere. We of course heeded that call and decided that we were going to implement this throughout our entire development organization. So it's been a pretty good story from there. Next slide. So what's really great about the fact that we have this installed in our CI pipeline, this is just a mock-up of [Away from mic] visualization tool on this slide is a couple things. One, we get line of sight into the actual activities of where we see failures across our development community. Probably the best benefit we get from that, I always look at what's my top five WCAG failures today, my top five WCAG failures of my products at workday would be different of somebody else's based on the unique attributes of your product. If I use that information to inform us, we can then recalibrate our training and have subsequent waves of training or improve our training over time to really improve those top five defects and see fewer and fewer of those. If I remeasure the top five defects a year later, hopefully I'll see a different top five. So the real big benefit that we see from having this is to be able to calibrate and really level up our development. The other cool thing that we've done is out of that trepidation of [Away from mic] barbarians at the gate is you can customize which WCAG failures will actually fail your build as you're trying to level up your organizations so you don't have to turn it on and everything fails, you can tune it based on where your organization is on your accessibility journey. And the last thing I'll share too and Bryan will take us through some more of this is the nice part about when you're using the DevTools enterprise version is all of this is exportable, comes out in JSON, this is our data visualization tool of choice but you can pull this into any data visualization tool you want and then drive that into the rest of your CI dashboards, if you will. I think Bryan is going to share with us the USAA perspective on this next.

>> Thanks Tom.

So when we looked at how do we scale at the enterprise from a USAA perspective, what we saw a few different things. First was, and Tom mentioned this, this is not specific to accessibility but this is in general the approach that I take. At USAA we have pretty much every programming language you can think of and I'm sure we have a few we've made up ourselves. Whenever I'm reporting my results to my test case management solution, we wanted to make sure we provide one API that reports all the test results regardless of what tool it is, et cetera. We mentioned we have the tools that Tom mentioned as well as many other ones. So we didn't want to tie it to an individual framework or individual test tool. Really as long as I report the results to our centralized API, then we can give the results and because we are a heavily regulated industry and organization, for us that's especially important because all these different tiers doesn't matter, it's important to show we're compliant and doing our testing properly and we have quality. That's the first thing. The next thing, what we wanted to see, and this is actually a learning I had myself, when I first came into the testing space and was trying to -- there's all these different various tools you can use for testing, what I said was hey, let's have people go to the dashboards that the tools provide. So tool A has a dashboard, tool B has a dashboard, you can build the awesome visualizations and let's just use those. What happened is I started discussing these conversations with some of our leaders and one of our top level executives basically said, Bryan, I'm going to look at one dashboard and that's it. So if the data is not on that dashboard, I'm not going to see it, I'm not going to look at it, eaten going to know about it. That was a very quick way for me to go great plan but that doesn't matter because they are not going to look at it. What we started to look at is how do we provide all the data on one pane of glass to make sure the information is there. When we look at the overall perspective of what that is, I'll give an example. At the top right what you will see is I have got something called number of test cases run. So you have manual and automated test cases. That's something standard you might see. Next one you might see is number of test cases run, I have got 12 failed, 8 incomplete, 160 passed, which provides you some amount of information. But what we want to start driving towards is on the bottom right which is here's the number of test cases I've run, but let me break it down into security, accessibility and performance so you can start understanding from that perspective, you have a high-level dashboard and you can dive into individual areas and say here's the number that passed and here's the number that's  failed. That's what we're trying to drive. You have to dive down deep into the information. Tom showed the example he gave. Maybe you dive down from this level and it will show the type dashboard he has. That's something to consider. Real quick, in the reference of time, when you think about accessibility testing, you have got automated and manual. Make sure you're pulling it into one test case management solution. And like I mentioned, regardless of what the testing tool is we might have essentially what we have on this is on this diagram we have accessibility test, performance test, functional test, security test, static code analysis, all of those flow into the test results like I mentioned before and go into Data Warehouse. Regardless of what tool it might be, let's make sure that we provide the API for that example. I'm going to have Tom talk next about catching regressions.

>> And maybe just a last story, promise. So after all of this it was really interesting, I was talking to one of our dev managers about why are we seeing numbers that are so low what we're seeing in the build pipeline?

He peeled that back and the real short version of the story is that most of our developers know they will fail or build so we're beating it in the [Away from mic] test so we're driving the right behaviors into our development community. We had one build fail a little while ago and the developer came to us and it wasn't like what do I do. It was like here's the error I got, here's the tool tip I got, I just wanted to run this by you, am I doing the right thing. I was like congratulations, you broke the app widget. And he was like what?

It's that space where you want to be, where they knew what's to be expected, they got the error and they just looked to us for advice and consult. That was to me a really great capstone of from the pilot to a couple months later applying to the entire development community in a sustainable and scalable way. So I think that wraps it up for what we wanted to present to everyone here today. I think that opens us up for Q&A, Ryan?

 

 

>> Yeah, Tom, Bryan, thanks so much. Fantastic. We have got a lot of chat. A lot of great questions. Let me start with the most popular question, which I think maybe both of you could weigh in on. It is:  Are there any accessibility training tools you recommend?

>> Training tools?

I think I'll answer this a little bit differently than what might be expected for the answer. I think many companies do this. We have what's called an accessibility research and training lab. It's a physical space in our office here in California but have since transitioned in the COVID era. There's lots of tools available to you actually at the Axe products available to you, lots of guides and how to's to help you understand the error and coding [Away from mic] those are good tools. I think we have taken the approach where training by role type in the development life cycle has been very helpful to us. It helps set the expectation of what's expected for the role and making it everybody's responsibility.

>> Yeah, I think I would say the same thing. I'm of the internet generation so I usually go to YouTube and try to find examples. That's usually a good way to do that. I've also done several different things within our organization we've got some great accessibility leaders. So we have even a lab where people can come in and learn about the tools and it may not scale as much, but it's a way to really help some of the leaders understand the value of, okay, how does trying to use this application work if I'm using a screen reader, for instance?

Those are good examples. Quite frankly, one thing I think is great about a lot of these tools is, again, a great way to start learning a tool is start using that tool. As it shows findings and shows issues, like I mentioned, a lot of these tools will say this is how this might be impactful. So if you start thinking about how if I used a screen reader, how I wouldn't be able to utilize this or if there's not captioning, et cetera, and it helps you understand the impacts of that, that can be a really good way to help it hit home for someone.

 

 

>> Thanks, Bryan. Yeah, a lot of good tools out there that will enable you to learn while doing,  right?

You touched on this a little bit. Maybe it's worth reiterating since it has a lot of up votes. This question is what tools are you -- I'm sorry. What tools are you making use of in your CICD pipeline and how?

Maybe we start with Bryan this time?

>> Sure. I can't go into like specific tools but I can tell you the capabilities that we have within our pipeline. So within our pipeline, obviously we have got unit testing, integration, we have got functional. We actually do security tests in the pipeline, performance testing and accessibility testing. So really what we've embraced with this and this is something I learned a long time when I was a developer, if you don't automate it or it's not within the main pipeline of work, it's probably not going to get done. So the most critical thing for us is make sure you're running those tools every time you do a check in from a CICD perspective or if you're not quite there yet, making sure you run those tools daily and teams are looking at it. Certainly we use all the tools within the pipeline.

>> Yeah. Maybe just to augment, when we were looking at the automation side of things too, sometimes there's the con argument that some of the accessibility automation tools only find 40% of accessibility issues. If you take accessibility out of the equation, I could tell you you could automate 40% of your testing, wouldn't you want to make the investment which is why we made the investment in  this. On the manual side of things there's obviously still that and obviously the Axe plug in and the unit test side of things and color contrast checker as  well. So we use the whole host of more traditional manual testing and obviously with some screen readers layered on top of that, we test for a couple of different [Away from mic]. Our workday, our clients are 44 million users of our products so we test for various permutations screen readers against different browsers is probably the most important side of what we test based on how our product is utilized.

>> That's a good point. One thing I forget to mention is let the machines do what they do best and let your experts find the things that are hard to find. Run half hour the host of tools there are, but run the automations first. Then once we get to the stuff you can't automate, that's when you have the experts and you still want to train, help them understand that, but utilize those tools as much as you can so you don't have to spend your time and money doing something that a machine can do.
>> Yeah. Thanks for that. There were quite a few questions in here asking about what y'all do with the manual testing component. Okay. Another popular question. How can someone start to advocate to include more people in accessibility training, not just more devs but product managers, designers, marketing, et cetera?
>> I'll tell you that I do a lot of road trips inside of my company and get in front of various parts of the organization. If I have an opportunity, I'm there to present, again with that mantra that it's everybody's responsibility. So one of the ways that we've done this at workday and many companies have, they may call them different things, we call them OKR, objectives and key results. We do these every 90 days, our goals against our high-level objectives. One of my pitches that I make and usually come out with a full class after telling people they should make this an OKR for the organization and they do. The OKR is to have everybody trained up on this. Just on the Axe DevTools I want to make sure every developer has been trained up on this. And we got 100% participation. I was shocked that we actually got that. It was because we made an OKR and people know they want to meet their goals and objectives so that's how workday pitched it that way.
>> I would add a couple different thoughts to that. One is be aware that so like I talked about quality that is everyone's responsibility. I have heard quality is everyone's responsibility, security is everyone's responsibility, all these things are everyone's responsibility. So be aware of it's my responsibility fatigue. So help people understand also, again, what's the personal impact, what's the personal value thaw see this, what happens -- so if you have an individual that utilizes those tools, have them explain potentially how it impacts them and their situation and their utilization of those tools. I know for me, that's how our leaders caught me, right, was hey, let's go talk about this, let's fully understand what this might be. For me, that was really helpful was actually understanding the personal side of things of how this impacts someone's daily life utilizing these tools. So I think that's important when you start understanding the implications of those tools and when something's not accessible, how really it can impede someone's ability to utilize your website. There's certain different ways. There's also the financial aspects that have as well. The community that utilizes those tools, but I think for me it was making it personal and understanding how important this is is really important.
>> Fantastic. Tom, Bryan, I think you have gifted us with a lot of great knowledge. I really appreciate both of your times. I want to thank everyone who took the time to view this and take away some notes and maybe start to put these things into action. Thank you for attending and Tom and Bryan, thank you so much.