Serving Everyone: Accessibility for Public Benefits

[amazonv]

 Serving Everyone: Accessibility for Public Benefits

Type: Breakout

Track: Design

Public benefits programs like Medicaid and SNAP must serve everyone, but too often treat accessibility as a to-do list item rather than a core value of service delivery. How should we define access for our neighbors with low incomes who have been and remain systematically oppressed, whose only way of accessing the internet is through a smartphone on a limited data plan? Join Code for America staff as they discuss their work with various state governments to build a social safety net that is truly accessible to everyone.

 

Notes

-don't overwhelm people with questions
-keep language simple - 5th grade
-mobile first most low income people only have a mobile phone
-links should be specific about what is next (i.e. not more information but information about X)

Closed Captioning
 

CAPTIONING PROVIDED BY:

ALTERNATIVE COMMUNICATION SERVICES, LLC

www.CaptionFamily.com

* * * * *

This is being provided in a rough-draft format.

Communication Access Realtime Translation (CART) is

provided in order to facilitate communication

accessibility and may not be a totally verbatim record

of the proceedings

 * * * * *

(it seems to be missing the start)
 

>> And most of all, it guarantees that the needs of people are put first. So when delivered  effectively, a human centered safety net has potential to make a transformative positive impact on people's lives. So today we are going to use the social safety net as an example of government services because it's the one that Deirdre and I work most closely in but it does extend to other government services and agencies and types, like, for example, the criminal justice system. Some of the principles here around human centeredness are around first many welcoming doors, providing equitable and positive experiences in both online and offline worlds. Second principle, that the system is easy to understand, that people should be able to make it through the process with minimal support. The third principle is around informed decisions that people should clearly understand implications of all the actions they take throughout the process. The fourth principle is that it be responsive to changing needs, that the system itself change based on people's real needs as well as shifts in policy and budget and in the world at large.

And the last principle is around simple actions, that each stage in the enrollment and eligibility process should be completed in as few steps as possible. So unlike the private sector, government has to serve everyone. In the private sector in commercial products or services, they can usually choose their customers. They will target the market. But government doesn't get to make that choice. And for good reason. Government has to and absolutely should serve everybody. And that means that government products and digital products like the topic of Axe-Con, have to be accessible. So they have to follow the accessibility standards because there's a mandate for them. So that means something like the revised updated section 508, it means WCAG 2.1AA and this is honestly really great for us as accessibility advocates working in the government space, that there are clear mandates, that there are accessibility professionals embedded within government who make sure that any tools, any web  tools, any digital tools that are published by the government are accessible. This isn't usually the case in many industries. It is the case in the government public sector and it is great that it is the case. We fill at things like the VPATs, the voluntary product accessibility template. We use Axe core and Axe tools to automate accessibility testing. Code for America even has its own accessible design system called honey crisp and that gives us a great accessible starting point that features accessibility as well as being really user friendly as well as mobile friendly. Government digital products have to be accessible. That's sort of the necessary but not sufficient case. The reason we say that is because government services also have to be accessible. So what do we mean by services?

In most states, for example, people can apply for public benefits in multiple ways. This gets to the many welcoming doors principle that I outlined  earlier.

So they can apply maybe through an online application, and yes, it definitely should be web accessible, accessible for blind, low vision, Deaf users. But they can also apply maybe through paper and mail it in or fax it in. They can apply over the  phone, so they may be able to call a call center, call a hotline and apply that way. And before and hopefully after the pandemic, they could apply physically at a human services office.

So our work is much more than digital. The government services base is much more than the digital realm. Services take place at all these other places, right?

So accessibility has to also extend beyond digital to these other places and spaces. So what does accessibility mean for a holistic, complex government service?

I'll handoff to Deirdre to talk about that.

>> Thanks, Luigi. We know that there are services outside of the digital ones that government is responsible for. And like many things, services don't just appear. They're designed!

So we want to introduce an idea to you all and see how you take it, but thinking of the service design, the design of these services as a principle of accessibility. So what does that actually mean?

Let's talk about what service design is. On the next slide we have the definition here from a book called:  This is service design doing. The way they define service design is service design is design for experiences that happen over time and across different touch points.

So this is interesting for us because we think about digital accessibility as one aspect of that but in all of these examples that Luigi gave of different ways that folks can start the benefits application journey, they are not all digital. What happens when you're at a physical office space?

What are the expectations when you call up to apply?

What are the ways that the paperwork is designed in such a way that is accessible for people?

So we want to consider all of those. So we won't -- on our next slide we laid out flat. We believe that when a service is truly accessible, it is accessible at every point in the process. So that's in all of these different ways. We have three main strategies that we are going to introduce now to sort of explain how we think about making services accessible at every point in the process. So on the next slide we have our first strategy. Our first strategy is no wrong door. When a service is truly accessible, it's accessible in many ways. So no matter where someone starts, that place should be the right place. For public benefits applications, that could be over the phone, in person, on the digital platform like Luigi was explaining. And when they actually get to that place, wherever they start, which is the right place, they should be able to speak their language. So it should be available in many languages.

 

 For public benefits in many states, the human services department contracts a language line and they can conference call with an interpreter so that no matter what language someone comes in speaking, they can be helped by the human services representative that's there, that staff member, even with staffing limitations. That's one way that states make it so there's no wrong door. And websites should always be available. They shouldn't have business hours. And you might think I'm being dramatic. What website has downtime now?

But many states' services departments do have planned downtime. For instance, Georgia gateway, which is a website to apply for SNAP, Medicaid is actually closed for planned downtime many times a week. In one instance we saw it was actually down for 13 hours. If you wanted to apply on Sunday morning, you would have to wait until 10:00 a.m. to use that website. And it's really tough because we expect that people are not going to be in the office all the time, call center staff may not always be available, but websites should be that 24/7 access point. So not only do they have to adhere to WCAG standards; they should also be available all times of day. That's our first strategy that when a service is truly accessible, it's accessible in every way.

Our second strategy is mobile first. So when a service is truly accessible, it also has to be accessible on every device. In the social services realm we really think about this quite a lot because we know that especially people who have low incomes, rely on mobile devices as their primary means of accessing the internet. According to peer Research Center about a quarter of low income people in the U.S. use a smartphone only to access the internet. If you design a website only available on a desktop, it's not really going to serve a vast group of people. And it's particularly resonant with people of color. We see these same kind of trends among all income levels for people who are Hispanic and Black. So we're thinking about the actual device itself, right?

We're thinking about a mobile device or tablet, we're thinking about internet access, but also the form function itself of a mobile device can be incredibly important when thinking about social services. For instance, part of the application experience that Luigi will go into a little bit more in a minute is uploading documents. So people will have to prove that they're eligible for a given program by submitting their pay stubs, for instance, or submitting a copy of their rental agreement. This can be really challenging when you have to mail it in, when you have to use a fax machine, when you have to bring it into the office. But digital is actually a great way to submit documents and we think of a mobile device as an ideal form factor for submitting documents. You have the document in front of you, snap a picture on your smartphone and upload it. These are ways we're thinking about mobile first as a strategy not only in the digital accessibility but the other functionality that mobile devices can have. We're sort of extending our understanding of accessibility beyond digital accessibility, per se, to a service, the entire service.

And the last strategy that I want to introduce is plain language, which is very familiar to this audience, I think. When a service is truly accessible, it can also be understood by the largest group of people. Language comes in in a variety of ways into a process of signing up for benefits and maintaining public benefits. So not only is there that initial application, there's the stuff before and after it.

So how do people learn about their eligibility for a given program?

How do people know which programs actually fit their needs?

How do people know what documents they need to submit, what the next steps in the program are, how they can maintain their benefits?

Language content is pretty much 100% of most benefits programs. So it has a huge impact on the way that we can make our services accessible. And so for us, that means that we write at a fifth grade reading level in all the materials we produce in our partnerships with states. We also design applications in a way that require minimal cognitive load. So that means asking as few questions as possible. For us that usually means one question per page. Remind me to tell you about this in a little bit when we talk about Minnesota. Luigi and I are on a project where we're designing a new benefits application for the State of Minnesota. We'll talk a little bit more about how this comes into play there. And also these questions that we write for states like Minnesota, we try to create them in such a way that they don't require mental gymnastics to be able to answer. I'm sure you've filled out forms that have a negative question with negative statements. We try as much as possible to make the question clear, concise and very easy to answer because we know that cognitive load is an issue with accessibility. And the last part about plain language is clear actions. One way we think about this with form design is providing options for buttons that actually tell you what the next step will be. So instead of saying next, we say continue on to a very specific next step that it will be, which is a better way to let people know what the outcome of their choices will be. So we really want to make sure that we're pairing whatever it is that we provide as an option going forward with a very clear outcome. So they always know what the outcome is going to be. To summarize, these are three main strategies that we think about on the next slide. We think about no wrong door, mobile first, and plain language. And we believe that using these three strategies in combination will actually be the practical steps to make sure our thesis true, our thesis being when a service is truly accessible, it's accessible at every point in the process. I've seeded this a little bit. I'm going to tell you a little bit more about the specific steps of the application and enrollment process to show you how we apply these strategies very specifically in our work.

>> Luigi:  This next slide shows the benefits journey. This is a journey map that we created from years and years of research in the public benefits space. I'm going to go over all the steps of the public benefits journey. So first it is around discovering benefits. So you hear about the benefit maybe, or you Google it. How do folks actually discover the benefits?

The second step is applying. We've already touched on this a little bit. Do you apply digitally?

Do you apply on paper?

Do you have help applying?

The third step of the journey is determining eligibility. This is usually happening at the government side, whether it's a caseworker or someone like that, a worker at the government puts in all your data from your application and determines whether or not you are eligible for that benefits program.

The fourth step is the notification that you are or are not eligible for the program.

The fifth step in the journey is actually using the benefits. So if it is food assistance, like the  SNAP program, that might mean using the EBT card, the debit card, at a grocery store. If it is a public benefits program like Medicaid it might mean using it at the doctor's office. The last step is maintaining benefits over the long-term. Many of these programs have renewal processes. The renewal can be anywhere from every three months to every six months to every year. So it really varies depending on the program. And at all those points, it's really important that we recognize that people can drop off, right?

  

  

 

We know a lot of people will fall off somewhere along this journey and they will have a hard time getting back on the program, especially at this last maintaining benefits step when people go off and on benefits, even though they're still eligible. That's called churn. There is churn in the program because maybe there's paperwork that's been missed or maybe a letter was just not mailed to the right address and people try to use their food stamps EBT card at the grocery store or they try to use their Medicaid at the doctor's office and they're denied. And that is a really hard situation and it requires the person on the public benefits program to really scramble and think about how they are going to resolve that situation. Let's talk about some of these steps in depth with some of the strategies that Deirdre previously talked about. I'm going to go to the next slide and talk about how people find out about benefits.

I think we can think of the no wrong door strategy that Deirdre talked about earlier. So if you were to apply for unemployment insurance, and maybe some of you have had to do so in the past year or so, or if you're applying for food stamps or any of the public benefits programs, what would you do?

Maybe you would search on Google. So there's search engines. Maybe you would hear about it from word of mouth. So you would have friends or family who know where to go. If you're digitally able, you would go to a government website if you have digital literacy. So you would find yourself on a government website. Then maybe you would have to figure out how to get to the actual application. That might be a little bit tough. You might learn about programs on social media. So we worked on a fantastic program called pandemic EBT in the states of California and Minnesota. And pandemic EBT was a program meant to fill in the gaps for school children who were on free and reduced lunch programs at their schools and who were missing out on those meals during the pandemic because schools were physically closed.

So the pandemic EBT program provided EBT cards, which are like debit cards, to families of these kids who could then buy groceries with those cards. And it was a brand new program created literally in months. And the way the word got out of that primarily was through social media, through school districts on their Facebook pages, on their email list serves getting the word out to parents that this program was happening and these are the steps to take to claim the benefits, which really, really helped families in need last year. People also find out about benefits through community-based organizations like food banks. So usually if folks go to food banks to get a box of  food, the food bank will also offer to help them and that family apply for programs like SNAP, like food assistance. So people find out about benefits in all these ways and we have to think about how to make all those different ways accessible.

Another step in the journey is around getting help. So people need to get help along the way. So there are, for example, multiple ways to ask for help and maybe different people prefer different modes of communication. So someone more digitally inclined might choose chat or SMS or text messages or email. They might prefer to communicate that way. Someone else might prefer to communicate over the phone, might want to talk to someone in real life. Other folks might prefer to go in person to digital services offices and unfortunately that hasn't been possible but hopefully will be possible again soon.

These services are complex. Most people who apply will report that they feel like they need to be an expert to navigate these systems. So for example, in Minnesota, as Deirdre mentioned earlier, we're working on an application for folks to apply to several public benefits programs. When people are applying, even though we have taken all these principles and strategies to heart, even though the website is very accessible, they still need help along the process. So we have installed a service called Intercom which is a live chat widget that goes on to any website. What we found is really just being able to get in touch with a real live person really helps people navigate the process, feel more comfortable with their answers, feel more confident that they are correctly applying for the benefit. And it's great to have another person at the other end to ask questions and to get answers and usually that person is my colleague, Deirdre, who is usually at the other end of that chat window.

So one thing around here, around this concept, around getting help is maybe after this try and see how you would get help. Maybe call, look up your local human services agency and just ask how would I apply for food stamps. What is that experience like?

I think you'll learn a lot. Another point along the journey, a very specific point is how people learn about their case status. So once people submit their application to a program, their number one question and frankly their number one source of anxiety is around the status of their case. When will they hear back?

Do they need to do anything else?

Has their case been lost in the system?

Here it's essential to set expectations with clear plain language about next steps. When to expect to hear back, when to know that maybe, oh, you should maybe call in. A case status usually unfortunately is not possible to find out for oneself. Here, again, multiple modes of communication are important. That no wrong door strategy really is important. As one example, we worked in the state of Louisiana a few years ago and our researchers looked into how people do this. How do people learn about their case status?

Turned out it was pretty much a rude Goldberg system. There was a call center, a hotline. If they called that call center, they would not get a real person at the other end. They would get a phone tree. They navigate the phone tree, they press three and then five and then seven and then one for a few minutes, maybe five minutes, maybe 10 minutes. Eventually they get to the right place, hopefully, which is a voicemail. That was the thing, you leave a voicemail with your request. So you have to hope at that point that you leave the right information, that you leave your name, date of birth, maybe social security number, maybe case number if you know it. You have to hope that when a call center worker listens to your voicemail some hours later, that they hear you well and that they can have all the information they need. From there the case center worker -- excuse me, the call center worker would actually write an email to a caseworker. So the call center worker did not have access to cases. Only a caseworker did. So they would then need to send the email with all your details to the caseworker. And then you have to hope that the caseworker can find your case and then call you back. If that was confusing, it's because it's a confusing process. And all of this was just a very complex system just to learn what's your case status. On the next slide we are going to start going through some case studies. So we are going to start on the next slide with Alaska. So we worked in Alaska a few years ago on a few pilots. In Alaska it's a very remote population, as you can imagine. Towns are  small. They're very rural. They're very far flung. There is low internet connectivity. So working with all these challenges around access, what did we do?

So we worked on the beginning of the benefits journey in Alaska. So here is another map. On this next slide there's another journey map and we're highlighting the first two steps of it where people can discover benefits and then they can apply for benefits.

So on the next slide for Alaska, we are going to talk about the Alaska fee agents system. So fee agents are Alaskans who help their rural neighbors apply for benefits. And on this slide there's a picture of a fee agent, so we went to Alaska and talked to several fee agents and here's one of them. She's showing us a paper application of the interview that she gives to folks, her neighbors really, who are applying for public benefits.

So this pilot was us essentially creating a prototype where we were converting this paper-only system where Alaskans could help their neighbors  apply. And the reason why this system exists, I should say, is because Alaska is so remote that there aren't human services offices in all these small towns. The government can't do that. So there are these local neighbors who act as fee agents who help out.

And the system was paper only. It was an interview process. The fee agent would usually sit down in the home of the person applying and talk to them and have a conversation about their situation and fill out this paper application with them.

So this was a guided application process. So I think that's really great. It's really great to have help when you're applying for public benefits. And it can be completed in concert with the applicants and the fee agent. And so we set out to make a prototype to digitize this process. So we were thinking about our strategies of no wrong doors, of mobile first. Well, in Alaska, there's low connectivity, right?

So especially in rural parts there's very low cell phone coverage specifically. With that, we designed our prototype to be offline available. So it was a mobile application. Could be run maybe on a tablet like an iPad, but it also had an offline mode so that data could be taken in and synced when there was connectivity again, maybe when the fee agent got back to WiFi or back to a less rural part that did have internet access.

So that was, again, thinking about how people are using the services and thinking about how we can build mobile first technologies that really don't have any wrong doors. Another prototype we did was around the case status problem I talked about earlier. This next slide has a picture of an Alaska woman holding up a paper prototype. This paper prototype looks like a web app, which talks about a tool called my Alaska and it is a log-in screen, but the only boxes on this log-in screen or this paper prototype are your name, the name on your case and your social security number and that's it. This is something that we'll talk about in a minute. It's a big issue around user accounts. And user accounts are really inaccessible. Instead, in this prototype we are talking about how can we identify someone using information they already know, their name, social security number, maybe date of birth. Making this log-in process easier and reducing burden on the end user. With that, I'm going to kick it off to Deirdre to talk about more case studies.

>> Cool. So I was telling you a little bit about Minnesota earlier. It's top of brain for Luigi and me because we have been working on this project for more than a year now actually with the State of Minnesota. And this project is an effort to build a new human centered, mobile first plain language application for folks in Minnesota to apply for SNAP, which are food stamps, and cash assistance programs and hopefully -- if you would like to click around yourself, you're welcome to. It's demo.benefits.org. And because it's a demo site you can click around without submitting any real information. Why did we do this in Minnesota?

Because Minnesota has one of the longest application times, sort of originated in a product that we did at code for America where we went to all 50 states and tried to apply for SNAP, cash programs and Medicaid. And if there was a digital option available, we tried to go through that entire application ourselves. And what we found is that the state of Minnesota, if you would like to apply for both SNAP and Medicaid in Medicaid, it will take the better part of two hours to do so, one of the longest in the country. The staff agreed that's not okay so we started a partnership to improve it. It's also really hard to find the applications. You might think, well, I'm experiencing some hardship, I need help paying for food and medical bills, why don't I just go online. There's this one department. There will be one application, right?

But not so!

There are two. And finding them is pretty challenging. So I'll echo Luigi's suggestion from earlier that you go for your state or the state of Minnesota and try to find the applications for these programs and it's no easy feat, I'll just give you a forewarning.

Let's talk about where we are in this process. In the next slide we're bringing up the same journey map that Luigi was showing you earlier with the next steps for applying and maintaining benefits. For the project in Minnesota we're focusing on the applying for benefits. Once you find the application, it doesn't get much clearer. On the next slide we'll talk you through the real problem that we see that actually stops people from beginning the application. And that was user accounts. Like I said, there are two different applications, one for Medicaid and one for SNAP and cash programs. On the right side of the screen we have a screenshot of the user account creation profile for apply MN, which is the SNAP and cash application for the state right now. People can't get through it. And this is an accessibility issue. You can make an application that's all to the top standards of digital accessibility, but if folks can't even start the application, it's a moot point. So this further reinforces our point that accessibility has to take place at every single step in the process. Our project right now doesn't require a user account to start. I want to just give you an example here of how this comes into play. On the next slide you'll see a screenshot. This is a picture of a client holding her cell phone. You can see the same website, Apply MN, on her phone. The type is incredibly small because it's not mobile first. It was designed to be used on a laptop or desktop computer. What this means is especially for clients relying on their cell phones to access the internet, this becomes a herculean task to try to actually start the application to find out where to begin. Looking at this screen, you see a big block of text in the middle. You think there might be some menus on the side, but there's no big digitally accessible button, no big tappable area that says start or apply. So it requires a lot of hunting and pecking to figure out where to even start.

It's a real problem. On the next slide you'll see the client was trying to do some problem solving here. Now she's turned the phone sideways. She has it in landscape mode and you'll see a big red bar. The beginning of the bar says the password entered is invalid. It goes on to explain all the password criteria. These are the characters you can use and these are the characters you can't. And none of this was provided as guidance in password creation on the previous screen where the client tried to actually create the password in the first place. Needless to say, this is really impossible. This client happened to have access to a desktop computer and she went on to that computer, went through the whole process and actually she got stuck inevitably again because her cookies were filling in a user account and password for an old application with the same Department of  Human services for a different program. So savvy person that she was, she went to her cookies and cleared them, something I wouldn't have even thought of, and was able to create a password and begin. This was the better part of 20 minutes where she was trying to start the application. And it doesn't take a great leap of imagination to understand why this is inaccessible. We know that we're seeing a couple of the strategies we talked about earlier. We're seeing mobile first come into play, we're seeing plain language, and these things have a huge impact on people's ability to get the help they need and deserve and pay for with their taxes. Last case study, we're nearing our end, we want to start with the work with Louisiana. In Alaska we have neighbors helping neighbors start the application process. Minnesota we see how just getting rid of username and password requirements lets people start the process. In Louisiana we were trying to tackle a different part of the benefits life cycle. On the next slide you'll see that same benefits journey and we are going to focus on two parts here. We're going back to part two, applying for benefits, and the last step in the process, part six, maintaining benefits. What do we mean when we say maintaining benefits?

All of these programs, in Medicaid, SNAP, cash assistance, clients not only need to apply to get  help, they need to fill out additional paperwork, maybe talk to a caseworker, maybe submit additional documentation at various touch points along the life cycle in order to maintain those benefits. So that could be for a SNAP client at six months they need to fill out a simplified report, pretty short, basically saying I'm in the same situation that I was six months ago when I initially applied; nothing's changed, I'm living where I'm living and earning what I'm earning. Or it might be at 12 months. We developed this idea of churn earlier. If you miss one of those things, you'll lose access to your benefits entirely and have to start from square one from the very beginning of the application process which is really onerous. In about 2018 and 19 we at code for America partnered with the state of Louisiana to try to intervene in some way to help people stay on benefits so they didn't have to reapply. So on the next slide you'll see a summary of that project. It was multi-program. So all the programs I was talking about, a one-way text messaging service where we broadcast reminders and guidance to clients at these key enrollment periods, these key points. On the right you'll see a picture here of two women and one of their very small babies who we were interviewing at a WIC clinic, women, infant and children, a program for women and children up to the age of five. We asked participants to explain what was going on with WIC in particular. Many of these clients were co-enrolled in other programs like Medicaid and SNAP as well so they had invaluable insights to share on what could help them stay on the benefits and make that life cycle of benefits be easier. What they said was every three months, at this time in Louisiana there were paper vouchers. So if you were participating in WIC in the women, infant children program, every three months you would come into the office and say yes, I'm still eligible for this program, here's my proof and you would get a piece of paper with your vouchers. As of 2019 all 50 states have EBT cards which is a huge difference. Needless to say, if you don't show up for those three-month appointments, you don't get the benefits. So showing up is really important. Up until our intervention, the only way that mothers, usually mothers, would know that they were due for an appointment is they would get a blue booklet at the end of their appointment. It would have a sticker on the back that says your next appointment is at this date and time. Now, imagine, you have this piece of paper, this folder. How many times have you lost a folder?

For me, many times. For many people, they didn't want to lose their folder so they put it in their purse that they carried around all the time, but the sticker was not the highest quality sticker and the date and time would rub off. It's really hard to remember when to come back. On the next slide, I want to show you what we ended up sending. We worked with the participants in this program and piloted a lot of these kinds of messages and got feedback, they told us change this word, change that, this is what would  help. I just want to read the two text messages on the slide. Three days before the appointment, we would send one reminder that said your WIC appointment is at 1/21 at 9:30 A&M at the office located at this  address. This appointment is to pick up your vouchers and will take about 30 minutes. You don't need to bring your child. Remember to complete your online nutrition class -- this was all this language that participants told us that would help, basically. When is where the appointment is taking place, what it's for, what they have to bring, and the consequence of not going to this nutrition class, so sort of incentivizing. The day before the appointment we sent a second reminder that's a truncated version saying this is a reminder that your WIC appointment is tomorrow at 9:30 a.m. again, it's when, where, why, what we need to bring and how to sort of problem solve if that wasn't going to work for folks. This was sent via text message. This is not a website, not an app. We weren't thinking about WCAG. What we were thinking about is the form itself. We're sending people messages where they are going to get them. How many people have their cell phones on them all the time?

Most. Most people have their cell phones on them a lot. And especially for programs like WIC, when thinking about pregnant women and young children, they tend to be younger than most of the population, cell phones are sort of ubiquitous. We're trying to not only think of accessibility in terms of the digital form but also thinking of it in terms of the mode.

So by now I hope we've proven our thesis, that when a service is truly accessible it's accessible at every point in the process. And that means getting into the product, navigating the product, all the stuff that takes place around it, people, places, and hopefully you'll think about this too when thinking about a definition of accessibility.

That's all we have got. I think we went a little bit over but hopefully we still have some time for  Q&A. All right. Great work, Luigi and Deirdre. Thank you so much for your presentation. I think we can sneak in one question before we have to wrap. So here's a question from Jessica. She asks when it comes to getting help with accessing public benefits, what are your thoughts on chatbots as opposed to real live chat agents?

Are there some areas where chatbots do a good job and do you have any advice for designing them with accessibility in mind?

>> Deirdre?

Would you like to take that?

>> Oh, would I?

  

  

I thought you might. That's interesting. Chatbots are a tricky one. I think we use some triaging. On our other product that we have at code for America, get cal fresh, we have zam also set up, this chat service. And we have a little bit of triaging at the beginning to help people get started with the conversation with a real human. What I find really challenging with chatbots specifically in the case of government services is it's so hard to get in touch with a real human and if we can provide a way for someone to find even me who doesn't work at the county but knows enough about where to send people, it's going to be a lot more human and a lot more comforting than a chatbot. So wherever we can intervene and create a human interaction where previously there was one that was more mechanical, I think the better. Particularly in the system when it's so hard to get in touch with someone and that's the only way to get answers. That said, if there's simple triaging, we want machines to be used for the things that machines are good at and humans to be used for the things that humans are good at. It's a balance to strike for sure.

>> Well said.

>> That's a great point. All right. Well, we are at our time now. Thank you everybody who submitted a question. Sorry we didn't get to them, but we want to thank Luigi and Deirdre again for an awesome presentation with tons of real world examples of what they're doing and how accessibility impacts it. So thank you so much to our presenters and to our guests, I hope everybody enjoys the rest of Axe-Con.