David Pratten is passionate about leading IT-related change projects for social good.
2801 stories
·
0 followers

From Milestones to a Continuous Quality Assurance Flow

1 Share

By Dr. Peter Fassbinder, Principal expert PLM Process Innovation, Siemens AG

One of the biggest hurdles to implementing continuous delivery and DevOps in an industrial environment—dealing with milestones/macroscopic quality gates—can be overcome with a continuous quality assurance approach that applies “green to green” and “shift left” paradigms to software and non-software artifacts.

In an industrial environment, software delivery requires much more than just working software. To formally release a change and deploy it into production requires many further non-software artifacts that cannot be covered by the continuous integration tool chain, no matter how sophisticated it is. However, using the classical milestone/macroscopic quality gate process is no longer an option, as this limits the deployment frequency to values that are outside the range of the desired target. Therefore, a rethinking of the release process is required.

This paper provides a new perspective on analyzing and tracking non-software quality criteria by introducing a continuous conformance concept. This concept applies the “green to green” and “shift left” paradigms—well established in the software space—to the non-software artifacts required to release and deploy a software change. Combined with a continuous delivery pipeline, this results in a continuous quality assurance approach that also enables industrial enterprises to take advantage of the many opportunities and benefits that arise from continuous enhancements of their products and services while maintaining their high quality standards.

The Vision—Continuous Flow of Value to Customers

With the boost of digital capabilities, industrial products are changing radically. Therefore, increasing efficiency is only one aspect of digitalization. The other, much more impactful aspect is the revolution of the customer experience. Transferring more and more functionality into software, combined with an ever-increasing connectivity to products at customer sites or in the cloud, opens up tremendous new possibilities for industrial enterprises and their customers. Continuous enhancements of products or cloud solutions during their lifetime, supported by data analysis from live systems, will become a key success factor.

This makes state-of-the-art approaches by IT companies, such as continuous delivery and DevOps, increasingly relevant for industrial enterprises as well. These approaches drive the vision of Agile all the way to the customer, resulting in:

  • faster value utilization and integration of customer feedback through short-cycled deployment of product enhancements
  • continuous, data-driven value increase due to product improvements based on operational data
  • reduced deployment and operational risks through highly automated delivery of small product changes

However, the required paradigm shift has a tremendous impact on the entire organization, especially in an industrial environment. Meeting the high industrial quality expectations in a world of continuous delivery is one of the key challenges that must be mastered in this context. This aspect is evaluated in detail in this paper.

Quality in the Continuous Delivery Pipeline

A core element and prerequisite for continuous delivery to customers (i.e., frequent deployment of software enhancements into the productive system) is the implementation of a continuous delivery pipeline. This integrated tool chain must be highly automated and should include automated tests on multiple integration levels, addressing functional and nonfunctional requirements. Once established, such a continuous delivery pipeline already provides huge benefits with respect to built-in quality mechanisms and real-time quality status information of the software. Typical quality information provided by a continuous delivery pipeline includes:

  • quality status of the build
  • test results on various levels, covering different test aspects
  • automated security checks
  • traceability from requirements over test cases to code and test results
  • continuous, automated, real-time visibility of all related status information
  • automated decisions about whether a build is a potential release candidate

In addition, this tool chain can be used to provide auditable evidence and track the status of risks and risk mitigations linked to the software artifacts. All of this is ideally made transparent in real time through a suitable dashboard providing a 360-degree status view on the quality of the software to be deployed into the productive system.

Therefore, an efficient and robust continuous delivery pipeline is a must to speed up deployment frequencies significantly—but it is not sufficient. Although such an automated tool chain already provides significant quality status information regarding the software itself, it lacks several aspects required for a formal release for deployment into the productive system in an industrial environment:

  • addressing strategic and system-level aspects, such as technical risk assessments, service level agreements, intellectual property rights, etc.
  • formal approval and release documentation, such as hazard summary reports, risk management reports, safety certification, etc.
  • further up-to-date, non-software artifacts required by different stakeholders, such as user documentation, operations guidelines, training, and many more

Before formally releasing and deploying a software change into the productive system, it must be ensured that those additional (non-software) artifacts are in a valid status and cover the change intended to be delivered to the customer adequately. These artifacts, however, cannot be covered by the continuous delivery pipeline, no matter how sophisticated it is. The continuous delivery pipeline is restricted to the software itself and software-close artifacts (e.g., requirements or test cases), but cannot cover the huge realm of additional artifacts required for a release, the non-software artifacts.

On the other hand, applying the classical milestone/macroscopic quality gate approach to ensure the quality of those non-software artifacts is also not an option anymore, as this would limit the deployment frequency to values that are outside the range of the desired target. Therefore, a rethinking of the overall release process is required.

“Green to Green” and “Shift Left” Are Not Only for Software

The most promising approach for a redesign of the classical release process into the “world of continuous” transfers a key idea of the continuous delivery pipeline to the non-software artifacts.

Within a fully automated continuous delivery pipeline, every code change is instantly verified against the existing system that was released in the previous iteration and is—at this particular time—running in the productive system. The primary focus of the continuous delivery pipeline is to verify that the code change does not break the running software, a paradigm referred to as “green to green.” With this approach, the quality of the software can be ensured for every code change at any point in time at the tip of a finger. Ensuring a system that is always running (i.e., a system that is “green”) has the utmost priority.

This “green to green” paradigm can be transferred to non-software artifacts. Assuming a minimal marketable product was already released to the customer at an earlier point in time, two things should be present:

  • a running software system in production
  • a valid baseline of all required, non-software artifacts matching the productive software version

Therefore, in addition to a green software status, there is also a green status for all required non-software artifacts at this particular time. Transferring the “green to green” paradigm from software to non-software artifacts implies that a suitable mechanism is now defined and established for the non-software artifacts, i.e., an analogy to a continuous delivery pipeline for the software. This mechanism, which can be labeled continuous conformance, is described in the next section.

However, to bring an adequate continuous conformance approach to life, another paradigm from the software world must be transferred to non-software artifacts: the “shift left” paradigm. This is required, in addition to the “green to green” paradigm, to ensure that relevant aspects of the non-software artifacts required for a release into production are addressed in a timely manner. The ambition is to achieve an operational capability where software can be seamlessly and continuously developed, integrated, tested, and deployed based on a highly automated continuous delivery pipeline without the need for lengthy and time-consuming work on non-software artifacts once the software is ready for deployment.

The “shift left” paradigm for software development ensures timely definition and early execution of tests across all development and integration stages to shorten feedback loops and speed up the capability for software delivery. By analogy, transferring the “shift left” paradigm to non-software artifacts has the intent of providing updated non-software artifacts matching the software version that will be altered through the intended software change exactly when needed. This means that non-software artifacts are analyzed and continuously updated at the right time to avoid lengthy and time-consuming work on non-software artifacts after the work on the software has been completed. How this can be put into practice as part of the continuous conformance concept is described in the next section.

The post From Milestones to a Continuous Quality Assurance Flow appeared first on IT Revolution.

Read the whole story
drpratten
7 hours ago
reply
Sydney, Australia
Share this story
Delete

A Solver of the Hardest Easy Problems About Prime Numbers

1 Share

In 2013, one of the best — but also one of the worst — things that can happen to a mathematician happened to James Maynard. Fresh out of graduate school, he solved one of the discipline’s oldest and most central problems, about the spacing of prime numbers. It was an achievement that ordinarily would have garnered him fame even beyond the cloistered world of pure math research. There was just one...

Source



Read the whole story
drpratten
7 hours ago
reply
Sydney, Australia
Share this story
Delete

DevOps vs. SRE vs. Platform Engineering? The gaps might be smaller than you think

2 Shares

Guest post originally published on the Humanitec blog by Luca Galante Platform engineering is the new cool kid on the block that everyone wants to be friends with. However, many are still confused where this new discipline comes from and how it differentiates from more established practices like DevOps and SRE. In this article, we provide some historical context and explain how they all relate to...

Source

Read the whole story
drpratten
3 days ago
reply
Sydney, Australia
Share this story
Delete

[Sponsor] Tailscale

1 Share

No additional hardware to manage. No complicated firewall rules. Free for personal use, but don’t take our word for it:

“Tailscale is the most insane piece of software I have ever used. It’s literally so good” —Twitter user @ennochian_

Link: tailscale.com/daringfireball?utm_source=df&utm_medium=RSS…

Read the whole story
drpratten
14 days ago
reply
Sydney, Australia
Share this story
Delete

OpenAI!

1 Share

I have some exciting news (for me, anyway). Starting next week, I’ll be going on leave from UT Austin for one year, to work at OpenAI. They’re the creators of the astonishing GPT-3 and DALL-E2, which have not only endlessly entertained me and my kids, but recalibrated my understanding of what, for better and worse, the world is going to look like for the rest of our lives. Working with an amazing team at OpenAI, including Jan Leike, John Schulman, and Ilya Sutskever, my job will be think about the theoretical foundations of AI safety and alignment. What, if anything, can computational complexity contribute to a principled understanding of how to get an AI to do what we want and not do what we don’t want?

Yeah, I don’t know the answer either. That’s why I’ve got a whole year to try to figure it out! One thing I know for sure, though, is that I’m interested both in the short-term, where new ideas are now quickly testable, and where the misuse of AI for spambots, surveillance, propaganda, and other nefarious purposes is already a major societal concern, and the long-term, where one might worry about what happens once AIs surpass human abilities across nearly every domain. (And all the points in between: we might be in for a long, wild ride.) When you start reading about AI safety, it’s striking how there are two separate communities—the one mostly worried about machine learning perpetuating racial and gender biases, and the one mostly worried about superhuman AI turning the planet into goo—who not only don’t work together, but are at each other’s throats, with each accusing the other of totally missing the point. I persist, however, in the possibly-naïve belief that these are merely two extremes along a single continuum of AI worries. By figuring out how to align AI with human values today—constantly confronting our theoretical ideas with reality—we can develop knowledge that will give us a better shot at aligning it with human values tomorrow.

For family reasons, I’ll be doing this work mostly from home, in Texas, though traveling from time to time to OpenAI’s office in San Francisco. I’ll also spend 30% of my time continuing to run the Quantum Information Center at UT Austin and working with my students and postdocs. At the end of the year, I plan to go back to full-time teaching, writing, and thinking about quantum stuff, which remains my main intellectual love in life, even as AI—the field where I started, as a PhD student, before I switched to quantum computing—has been taking over the world in ways that none of us can ignore.

Maybe fittingly, this new direction in my career had its origins here on Shtetl-Optimized. Several commenters, including Max Ra and Matt Putz, asked me point-blank what it would take to induce me to work on AI alignment. Treating it as an amusing hypothetical, I replied that it wasn’t mostly about money for me, and that:

The central thing would be finding an actual potentially-answerable technical question around AI alignment, even just a small one, that piqued my interest and that I felt like I had an unusual angle on. In general, I have an absolutely terrible track record at working on topics because I abstractly feel like I “should” work on them. My entire scientific career has basically just been letting myself get nerd-sniped by one puzzle after the next.

Anyway, Jan Leike at OpenAI saw this exchange and wrote to ask whether I was serious in my interest. Oh shoot! Was I? After intensive conversations with Jan, others at OpenAI, and others in the broader AI safety world, I finally concluded that I was.

I’ve obviously got my work cut out for me, just to catch up to what’s already been done in the field. I’ve actually been in the Bay Area all week, meeting with numerous AI safety people (and, of course, complexity and quantum people), carrying a stack of technical papers on AI safety everywhere I go. I’ve been struck by how, when I talk to AI safety experts, they’re not only not dismissive about the potential relevance of complexity theory, they’re more gung-ho about it than I am! They want to talk about whether, say, IP=PSPACE, or MIP=NEXP, or the PCP theorem could provide key insights about how we could verify the behavior of a powerful AI. (Short answer: maybe, on some level! But, err, more work would need to be done.)

How did this complexitophilic state of affairs come about? That brings me to another wrinkle in the story. Traditionally, students follow in the footsteps of their professors. But in trying to bring complexity theory into AI safety, I’m actually following in the footsteps of my student: Paul Christiano, one of the greatest undergrads I worked with in my nine years at MIT, the student whose course project turned into the Aaronson-Christiano quantum money paper. After MIT, Paul did a PhD in quantum computing at Berkeley, with my own former adviser Umesh Vazirani, while also working part-time on AI safety. Paul then left quantum computing to work on AI safety full-time—indeed, along with others such as Dario Amodei, he helped start the safety group at OpenAI. Paul has since left to found his own AI safety organization, the Alignment Research Center (ARC), although he remains on good terms with the OpenAI folks. Paul is largely responsible for bringing complexity theory intuitions and analogies into AI safety—for example, through the “AI safety via debate” paper and the Iterated Amplification paper. I’m grateful for Paul’s guidance and encouragement—as well as that of the others now working in this intersection, like Geoffrey Irving and Elizabeth Barnes—as I start this new chapter.

So, what projects will I actually work on at OpenAI? Yeah, I’ve been spending the past week trying to figure that out. I still don’t know, but a few possibilities have emerged. First, I might work out a general theory of sample complexity and so forth for learning in dangerous environments—i.e., learning where making the wrong query might kill you. Second, I might work on explainability and interpretability for machine learning: given a deep network that produced a particular output, what do we even mean by an “explanation” for “why” it produced that output? What can we say about the computational complexity of finding that explanation? Third, I might work on the ability of weaker agents to verify the behavior of stronger ones. Of course, if P≠NP, then the gap between the difficulty of solving a problem and the difficulty of recognizing a solution can sometimes be enormous. And indeed, even in empirical machine learing, there’s typically a gap between the difficulty of generating objects (say, cat pictures) and the difficulty of discriminating between them and other objects, the latter being easier. But this gap typically isn’t exponential, as is conjectured for NP-complete problems: it’s much smaller than that. And counterintuitively, we can then turn around and use the generators to improve the discriminators. How can we understand this abstractly? Are there model scenarios in complexity theory where we can prove that something similar happens? How far can we amplify the generator/discriminator gap—for example, by using interactive protocols, or debates between competing AIs?

OpenAI, of course, has the word “open” right in its name, and a founding mission “to ensure that artificial general intelligence benefits all of humanity.” But it’s also a for-profit enterprise, with investors and paying customers and serious competitors. So throughout the year, don’t expect me to share any proprietary information—that’s not my interest anyway, even if I hadn’t signed an NDA. But do expect me to blog my general thoughts about AI safety as they develop, and to solicit feedback from readers.

In the past, I’ve often been skeptical about the prospects for superintelligent AI becoming self-aware and destroying the world anytime soon (see, for example, my 2008 post The Singularity Is Far). While I was aware since 2005 or so of the AI-risk community; and of its leader and prophet, Eliezer Yudkowsky; and of Eliezer’s exhortations for people to drop everything else they’re doing and work on AI risk, as the biggest issue facing humanity, I … kept the whole thing at arms’ length. Even supposing I agreed that this was a huge thing to worry about, I asked, what on earth do you want me to do about it today? We know so little about a future superintelligent AI and how it would behave that any actions we took today would likely be useless or counterproductive.

Over the past 15 years, though, my and Eliezer’s views underwent a dramatic and ironic reversal. If you read Eliezer’s “litany of doom” from two weeks ago, you’ll see that he’s now resigned and fatalistic: because his early warnings weren’t heeded, he argues, humanity is almost certainly doomed and an unaligned AI will soon destroy the world. He says that there are basically no promising directions in AI safety research: for any alignment strategy anyone points out, Eliezer can trivially refute it by explaining how (e.g.) the AI would be wise to the plan, and would pretend to go along with whatever we wanted from it while secretly plotting against us.

The weird part is, just as Eliezer became more and more pessimistic about the prospects for getting anywhere on AI alignment, I’ve become more and more optimistic. Part of my optimism is because people like Paul Christiano have laid foundations for a meaty mathematical theory: much like the Web (or quantum computing theory) in 1992, it’s still in a ridiculously primitive stage, but even my limited imagination now suffices to see how much more could be built there. An even greater part of my optimism is because we now live in a world with GPT-3, DALL-E2, and other systems that, while they clearly aren’t AGIs, are powerful enough that worrying about AGIs has come to seem more like prudence than like science fiction. And we can finally test our intuitions against the realities of these systems, which (outside of mathematics) is pretty much the only way human beings have ever succeeded at anything.

I didn’t predict that machine learning models this impressive would exist by 2022. Most of you probably didn’t predict it. For godsakes, Eliezer Yudkowsky didn’t predict it. But it’s happened. And to my mind, one of the defining virtues of science is that, when empirical reality gives you a clear shock, you update and adapt, rather than expending your intelligence to come up with clever reasons why it doesn’t matter or doesn’t count.

Anyway, so that’s the plan! If I can figure out a way to save the galaxy, I will, but I’ve set my goals slightly lower, at learning some new things and doing some interesting research and writing some papers about it and enjoying a break from teaching. Wish me a non-negligible success probability!

—————————

Update (June 18): To respond to a couple criticisms that I’ve seen elsewhere on social media…

Can the rationalists sneer at me for waiting to get involved with this subject until it had become sufficiently ”respectable“ and ”mainstream” and ”high-status”? I suppose they can, if that’s their inclination. I suppose I should be grateful that so many of them chose to respond instead with messages of congratulations and encouragement. Yes, I plead guilty to keeping this subject at arms-length until I could point to GPT-3 and DALL-E2 and all the rest to justify the reality of the topic to anyone who might criticize me. It feels like I had principled reasons for this: I can think of almost no examples of research programs that succeeded over decades even in the teeth of opposition from the scientific mainstream. If so, then arguably the right time to get involved with a “fringe” scientific topic, is when and only when you can foresee the path to it becoming the scientific mainstream. At any rate, that’s what I did with quantum computing, as a teenager in the mid-1990s. It’s what many past scientists did with (e.g.) the prospect of nuclear weapons. And if you’d optimized for getting the right answer earlier, you might’ve had to weaken your filters and let in a bunch of dubious worries that you shouldn’t have. But I admit the possibility of self-serving bias here.

Should you worry that OpenAI is just hiring me to be able to say “look, we have Scott Aaronson working on the problem,” rather than actually caring about what its safety researchers come up with? I mean, I can’t prove that you shouldn’t worry about that. In the end, whatever work I do on the topic will have to speak for itself. For whatever it’s worth, though, I was impressed by the OpenAI folks’ detailed, open-ended engagement with these questions when I met them—sort of like how it might look if they actually believed what they said about wanting to get this right for the world. I wouldn’t have gotten involved otherwise.

Read the whole story
drpratten
16 days ago
reply
Sydney, Australia
Share this story
Delete

How to play with the GPT-3 language model

2 Shares

I ran a Twitter poll the other day asking if people had tried GPT-3 and why or why not. The winning option, by quite a long way, was "No, I don't know how to". So here's how to try it out, for free, without needing to write any code.

You don't need to use the API to try out GPT-3

I think a big reason people have been put off trying out GPT-3 is that OpenAI market it as the OpenAI API. This sounds like something that's going to require quite a bit of work to get started with.

But access to the API includes access to the GPT-3 playground, which is an interface that is incredibly easy to use. You get a text box, you type things in it, you press the "Execute" button. That's all you need to know.

How to sign up

To try out GPT-3 for free you need three things: an email address, a phone number that can receive SMS messages and to be located in one of this list of supported countries and regions.

  1. Create an account at https://openai.com/join/ - you can create an email/password address or you can sign up using your Google or Microsoft account
  2. Verify your email address (click the link in the email they send you)
  3. Enter your phone number and wait for their text
  4. Enter the code that they texted to you

New accounts get $18 of credit for the API, which expire after three months. Each query should cost single digit cents to execute, so you can do a lot of experimentation without needing to spend any money.

How to use the playground

Once you've activated your account, head straight to the Playground:

https://beta.openai.com/playground

The interface looks like this (it works great on mobile too):

A heading says Playground. There is a text area with "Write a tagline for an ice cream shop" in grey, and a green Submit button. A right hand panel includes some sliders and other options.

The only part of this interface that matters is the text box and the Submit button. The right hand panels can be used to control some settings but the default settings work extremely well - I've been playing with GPT-3 for months and 99% of my queries used those defaults.

Now you can just type stuff into the box and hit that "Submit" button.

Try this one to get you started:

Three reasons to start a succulent garden

The same interface. I have entered the prompt "Three reasons to start a succulunt garden". GPT-3 has replied, its output in the same text area but highlighted with a green background: "1. Succulents are low-maintenance: They don't require much watering or fertilizing, and they can tolerate a wide range of light conditions. 2. Succulents are drought-tolerant: They're perfect for areas that receive little rainfall or irrigation. 3. Succulents add interest and variety to the landscape: With their wide range of shapes, sizes, and colors, they can provide a unique and eye-catching addition to any garden."

Prompt engineering

The text that you entered there is called a "prompt". Everything about working with GPT-3 is prompt engineering - trying different prompts, and iterating on specific prompts to see what kind of results you can get.

It's a programming activity that actually feels a lot more like spellcasting. It's almost impossible to reason about: I imagine even the creators of GPT-3 could not explain to you why certain prompts produce great results while others do not.

It's also absurdly good fun.

Adding more to the generated text

GPT-3 will often let you hit the Submit button more than once - especially if the output to your question has the scope to keep growing in length - "Tell me an ongoing saga about a pelican fighting a cheesecake" for example.

Each additional click of "Submit" costs more credit.

You can also add your own text anywhere in the GPT-3 output, or at the end. You can use this to prompt for more output, or ask for clarification. I like saying "Now add a twist" to story prompts to see what it comes up with.

Further reading

Read the whole story
drpratten
20 days ago
reply
Sydney, Australia
Share this story
Delete
Next Page of Stories