QA Detective Story: Tom Hertz, the first Test PI, and the transaction failure mystery
It was another day to pass. I sat in my dark, windowless office. I glanced at the wall clock, only part of which could be seen among all the files, cartons, and other objects that had to be arranged someday. One day that will happen too, I thought to myself.
On the table, my good friend Johnny winked. It was only 10:30 in the morning, but a friend is a friend, and Johnny Walker is a real friend who does not leave you in good and bad.
I looked at the door. You could see through the glass that was part of the door a caption that read "Tom Har z, privat test d tective." It must be those bastard kids who erased the letters.
Johnny was already in the air, looking for a glass to share the forgetfulness with me when a knock was heard in the room. At first, I did not relate, because you do not have to help the creditors. But a velvet voice, that did not sound like Igor or Joe from the check discount company, echoed in the room like waves of a Jacuzzi: "Is there anyone here? Tom Hertz?"
I immediately hid Johnny, hurled some embarrassing objects into cardboard boxes, gathered all nonchalant I had left and said, "It's opened, you can enter."
It was as if the room had been lit with smart LED lights that communicated with my heart through Bluetooth. Into the room came the most beautiful woman I had seen, standing erect in a black dress that revealed more than it was hiding, wearing sunglasses.
"Come in, come in," I said to her, removing a few bags from the guest chair.
She looked for a moment in the room, then in the chair, and did not move. "Wait, second," I said, and found a towel that looked clean and put on the chair.
"The cleaning woman is ill this week," I said.
"Are you sure she did not drown in Titanic?" She asked, but sat down.
"how can I help?" I asked.
She took off her sunglasses and revealed green eyes that made you want to do whatever it took to make them look at you affectionately. I tried to concentrate.
"My name is Shani Penn," she said, managing projects in a start-up on stealth mode, but is already very invested. It's supposed to be a platform for a targeted store that will compete even with Amazin. There are investments in hundreds of millions of dollars, and senior figures in the Israeli economy are behind it."
"Sounds good," I said. "So why are you here?"
"Because its launch is supposed to be in three days, all the preparations, including the publication, are ready, and this is a date that we have committed to and to change it will be a disaster." She stopped for a moment, took a deep breath, and went on. "We have a problem," she said, "we discovered that there were orders that simply did not end properly.
"And why haven't you done code analysis again to find the root cause?" I asked.
"Because the tests, as always, only found out it at the last minute, that is, yesterday, and we do not have time, we need the most professional expert to find the problem ASAP."
Did I hear some sarcasm in her words? I asked myself, but I replied, "I am your man. It will cost you $ 100 a day + expenses."
"I'll give you even 500, just save us!" Said Shani desperately.
"No problem," I said aloud and to myself, how the hell can I solve this mess?
An hour later, I sat with Sharon the tester. "What is the scenario?" I asked. "After the purchase button is pressed. First, you choose a product, press it and go to a page with the details of that product, click on "Buy", and after you are charged you return to the page of the product without receiving notice of purchase. The problem is that the action of buying is not listed in the database and therefore the product is not sent, but you are nevertheless charged. "Listen," Sharon continued, "I don't know what the programmers will tell you, but they changed something in the latest version we got yesterday morning. They claim that only a small bug related to UI was fixed, but we know they can not be trusted!" "Not at all," said another tester who sat in the room and felt that he, too, had to say something. "So do you think they put in a new code that caused the problem?" I asked. "Sure, otherwise how do you explain that we did the same tests we do all the time, and only now have we found the bug?" "Only a change of code," said the second tester with finality. "Are you sure?" I asked. "Maybe you did something different - a new test, new values?" "Yeah, sure we did," replied Sharon sarcastically. "Sure," answered the shadow. "Go to Mary from the backend, sais Sharon. I'm sure she did something, I know her," whispered Sharon. "Yes, Mary," said the echo. "Thank you," I said, and went to Mary.
"Listen," said Mary, "I know exactly what the problem is, thank God. The transaction fails just after I get JASON on the REST API sent to me from the frontend. I do not know exactly what it is in the JASON, but the problem is that the frontend code is like what they like to eat: spaghetti. Any small change here will break something in a different zone. They have no idea what's going on there, so it's convenient for Jason to accuse me, even code review is not conducted there," she said dismissively. "The JASON blames you?" I asked in amazement. "Jason, well, he's the team leader of the frontend."
A minute later I'm at Jason's.
"Hey," said the bearded guy with the flowered shirt and round sunglasses. "What's going on, you want to drink? I have good stuff here, juice with ground chickpeas and another plant that I can not share its name." "No thank you," I said. "Tell me about the problem you have in the flow of the purchase." "Unfortunately, though I wish I could not help," Jason said. "Listen, we did not change anything, so it could just be the backend, we send JASON exactly as it says in Jira, if they do not know how to publish it, their problem, you know they do not even have a code review?"
I realized that out of these clowns I will not get anything that will help me to progress, and I asked Shani where the DBA was.
"In the opposite building. Go down through the parking lot, it's the easiest," she said. On the way, I thought the problem was probably the JASON. It makes sense, according to the scenario. The fault occurs after it reaches the server.
I did not get to the DBA. When I walked in the driveway I felt that someone had switched off the lights, like a power-off server without UPS. When I woke up I was on the floor with a terrible headache. Someone in a stocking hat stood over me. "Leave this investigation, clear?" He kicked me to sharpen the "clear" point and left.
I went back to Shani who took care of the ice and the towel which she held over the bump in my head. "You'll be fine," she said. "But if you choose not to continue, it is also okay." "I'm not stopping," I told her.
"Well, if there is ice, wait a second." She disappeared and returned with Chivas Regal. "He is 12 years old," she said. "No thank you," I said, barely restraining myself. "I'm on duty." "You're really brave," she said, patting my face, smiling and walking away. I felt like an autonomous vehicle that recognized an abyss where there was a highway, but I caught myself and called Sasha, the DBA. Let him come here this time.
"The data on Big Data is fully tested and fully integrated, the ETL works beautifully, the Spark stands out, Kafka works almost without using CPU, the Elastic is flexible and the reports are divine." Sasha The DBA fell silent. "And ..." I asked. "I do not know," Sasha replied, "I'm out of buzzwords. "And are you really using all this technology?" "Sure, yeah." "Can I see the servers of the Elastic, or at least connect with them?" A mumble. "Sorry, I did not understand, can you repeat what you said?" I asked. "They're not active yet," he said, looking in the direction of my shoes. "So, how does it work? What happens when I press Buy?" "When I click Buy, the request, which includes parameters such as product name, ID, price, etc., is sent to the server as a JASON, where it is parsed, and then a query is built and the information is inserted into the database." "And the payment details are actually transferred to the clearing department, after registration in the database?" "No, at the same time, that's how we save time." "Who came up with this idea?" "The Architect". "Who is the Architect?" "Me too." Ain't that grand.
I went back to the frontend. "What do you send to the backend when someone presses the Buy button?" "When the customer clicks on a purchase, a JASON is sent to backend, with all the details of the customer including a payment order, and a lot of details including your location, the time at the client (if available), if you are registered, if you have products in the basket and such."
I went back to the backend and asked, "When you get the JASON after pressing Buy, do you validate the data?" "With God's help," Mary replied. I thought she might not. Maybe she's just validating the JASON.
I had an idea. I went to Jira to check the requirements, opened Fiedler, and did some shopping in the testing environment to confirm my theory.
It was nine o'clock when I set a meeting with all the parties.
Everyone came, even the CEO. "I know exactly who is to blame and soon you will know. "You, Sasha, are a great architect. It is customary for a transaction to take place only when all the steps in it has passed." "Yes, of course," he replied, looking nervous. "But management pressed for speed, at least until we get to implement the Big Data ..."
"And you, Sharon, are sure of the tests you're doing?" "Sure," he said. "The most professionalism there is." "And you have done API testing?" "I do not have to," he said. "If the transaction works, everything is good." "But there is a problem and the transaction did not work." "It's not connected, the system knows what I want to buy." "Is that so"? I asked rhetorically. "You do not even know what the exact scenario is."
"Mary, you are validating the JASON, not the values. Say, after inserting the values to the database, do you check that the date was saved well? " "No, why what could go wrong?" "And here it went wrong."
"And you, Jason, before you send the data to the server, do you check that everything according to the requirements?" "Listen, brother, the server verifies." "Always double check, there may be mistakes the server will miss." "True, they are always wrong." "No, you are a righteous man," cried Mary. I silenced them.
"I'll explain what happened," I said. "One of the parameters is whether the product is in the basket when you buy it, the values are supposed to be binary, 1 or 0. In the last version, there is one change, when the product is not in the basket, instead of zero, nothing was written. So the parser instead of ignoring the parameter and moving on, it simply puts in the same parameter in the next value. If the parameter was &cart= and the next parameter is pname=shoe101. Because of the bug, 'shoe101' value inserted into the cart column. The result is cart=shoe101." And Sasha continued, "but the 'cart' is defined in Data Base as a binary column and shoe101 is text, it will never work..." "Well done, Einstein," I told him. "This is exactly the problem, the database does not enter the record, no order is issued, but the customer is charged. You should check the logs sometimes, BTW."
Everyone applauded me. Mostly Shani. "I always knew you would succeed," she said, hugging me. I almost got confused, but only for a moment. "Not really," I told her. "what?" She asked, and everyone fell silent. "You brought me because I'm not really functioning, but suddenly I started to move in. You got scared and hired thugs to beat me." "You'll never prove it, and why should I do it?" "Because you are an Amazin agent, of course, and you have received an order to hurt this company. I can't prove, but I saw in LinkedIn that you have a past as a programmer. You entered and changed the code: in case the product is not in the basket, you replaced '0' it with "". I checked in the GIT, it was on a day when Mary was not at work at all. Shani lost her ability to speak, and looked dangerously pale, like a dog running after a cat and the cat suddenly turning and then the dog freezes with fear.
When the shouting began, I walked quietly to Shani's room, took the whiskey that remained and went out into the street.