Confidence intervals, Confidence levels and probability of simple tests

It seems to be a simple problem, but i cant figure it out

Lets say, I would like to know if there is some point to implement new feature. If we have to focus on the feature or not. Lets assume there is no possible some kind of test like questioning the users or whatever else. Its function will be easy, something like – for example “webcam for ecommerce for users that are paying premium account”.

To be specific, I have 1500 premium users. I can tell “Feature is used when atleast 75% of clients use it”. Great! We would like to run Fake Door test, where we implement just the button for webcam and when user click on it, we show him “we are implementing this feature right now, stay with us” or whatever else (i know, fake doors isnt the best method, but it is not the point of this). I will “test” it for 14 days. In 14 days, 350 clients will come on my site and they see this feature. 265 of clients clicks on the button.

What can I say about this feature? It seems like I can say “Yes, we have to implement it, because 75% of users will use this feature” (75% of 350 is 262.5 < 265) => H0 (Atleast 75% use this feature) seems to be ok. But it is not truth at all. Because there can be HUGE error (I tested ONLY around 23% of clients).

What I am trying to achieve is:
I would like to say – “With 95% confidence, 75% of clients will use this function, so we can implement it”.

I am lost of all confidence intervals, confidence levels and sample sizes, etc etc. Can someone help me how to get the confidence step-by-step and explain me, what can I count from those numbers (1500 premium users at all, 350 users saw the feature, 265 users used the feature).

What are some real world examples of great unit tests? Links to Github repos welcome

I think the title says it all. I’m learning how to write better unit tests and I think that a great way to do it would be to look at some real, production sofware that has really good unit test written. Can you recommend some examples? The language doesn’t matter.

What events, tests, and troubles could a party trying to establish a nation expect to face

My players have decided to establish a new nation which consists of Phandalin, Triboar, Red Larch, and all lands in between (per their own highly unofficial declaration). I’m trying to come up with some things to present these new kings and queens that will test their abilities to govern well. Anyone have any ideas mechanics to use to test them without getting too bogged down with the less exciting aspects of being a politician.

A few details that might help:

  • The party is level 12
  • They are a generally chaotic good party consisting of a Rogue,
    Sorcerer, Druid/Ranger, and Rogue/Warlock

  • The campaign started with Lost Mines of Phandelver and transitioned
    into a slightly homebrewed version of Princes of the Apocalypse

  • There is a zombie apocalypse which has destroyed all known
    civilization south of Waterdeep which has created a huge refugee
    issue in the North.

  • Neverwinter recently got infected with the plague and the party
    kidnapped/rescued King Neverember. He’s currently a political
    hostage.

  • Lords Alliance is the main government entity in the region. They used the Witch Hunters, Neverwinter, and Waterdeep as their military forces.

  • The reason the nation was established is because the party freed
    Triboar from the rule of “The Witch Hunters” who had taken it over to create themselves a headquarters from which to hunt down and kill or imprison all Arcane and Nature magic users. So when the party cut the head off the snake they didn’t want to let the locals fall into
    chaos, especially with all of the plague and refugee stuff going on.

How exactly should unit tests be written without mocking extensively?

As I understand, the point of unit tests is to test units of code in isolation. This means, that:

  1. They should not break by any unrelated code change elsewhere in the codebase.
  2. Only one unit test should break by a bug in the tested unit, as opposed to integration tests (which may break in heaps).

All of this implies, that every outside dependency of a tested unit, should be mocked out. And I mean all the outside dependencies, not only the “outside layers” such as networking, filesystem, database, etc..

This leads to a logical conclusion, that virtually every unit test needs to mock. On the other hand, a quick Google search about mocking reveals tons of articles that claim that “mocking is a code smell” and should mostly (though not completely) be avoided.

Now, to the question(s).

  1. How should unit tests be written properly?
  2. Where exactly does the line between them and integration tests lie?

Update 1

Please consider the following pseudo code:

class Person {     constructor(calculator) {}      calculate(a, b) {         const sum = this.calculator.add(a, b);          // do some other stuff with the `sum`     } } 

Can a test that tests the Person.calculate method without mocking the Calculator dependency (given, that the Calculator is a lightweight class that does not access “the outside world”) be considered a unit test?

Struggling with cyclical dependancies in unit tests

I’m trying to practice TDD, by using it to develop a simple like Bit Vector. I happen to be using Swift, but this is a language-agnostic question.

My BitVector is a struct that stores a single UInt64, and presents an API over it that lets you treat it like a collection. The details don’t matter much, but it’s pretty simple. The high 57 bits are storage bits, and the lower 6 bits are “count” bits, which tells you how many of the storage bits actually store a contained value.

So far, I have a handful of very simple capabilities:

  1. An initializer that constructs empty bit vectors
  2. A count property of type Int
  3. An isEmpty property of type Bool
  4. An equality operator (==). NB: this is a value-equality operator akin to Object.equals() in Java, not a reference equality operator like == in Java.

I’m running into a bunch of cyclical dependancies:

  1. The unit test that tests my initializer need to verify that the newly constructed BitVector. It can do so in one of 3 ways:

    1. Check bv.count == 0
    2. Check bv.isEmpty == false
    3. Check that bv == knownEmptyBitVector

    Method 1 relies on count, method 2 relies on isEmpty (which itself relies on count, so there’s no point using it), method 3 relies on ==. In any case, I can’t test my initializer in isolation.

  2. The test for count needs to operate on something, which inevitably tests my initializer(s)

  3. The implementation of isEmpty relies on count

  4. The implementation of == relies on count.

I was able to partly solve this problem by introducing a private API that constructs a BitVector from an existing bit pattern (as a UInt64). This allowed me to initialize values without testing any other initializers, so that I could “boot strap” my way up.

For my unit tests to truly be unit tests, I find myself doing a bunch of hacks, which complicate my prod and test code substantially.

How exactly do you get around these sorts of issues?

How to write DRY Unit Tests when order of method calls matters

Suppose I’m writing a calculator engine with a class as follows…

public class ArithmeticExpression {     ArithmeticExpression Add(double operand) { /* ... */ }     ArithmeticExpression Subtract(double operand) { /* ... */ }     // ... other useful operations like division, multiplication, etc.     double ProduceResult() { /* ... */ } }  

… and the goal is to defer evaluation of the expression until ProduceResullt() is called, so that correct order of all stacked operations is maintained.

My question is:

How should I tackle with unit testing this class without getting overwhelmed by the number of combinations of method calls?

It is clear to me that the order of calls should be tested somehow, but writing tests like this:

public void Multiply_CalledAfterAdd_TakesPrecedenceOverAdd() {      // ... } public void Multiply_CalledAfterDivide_EvaluatesLeftToRight() {     // ... } 

can get maddening and out of control quite quickly. And it’s not even about those 16 tests for the four basic arithmetic functions. Suppose in the future I decide to extend the calculator with a modulo operation – that’s 9 almost identical tests more to begin with.

On the other hand – the whole puprose of the class is to keep the order of operations maintained, so it’s got to be verified somehow.

Wrapping up – I know the example is trivial, but it’s a general problem that I struggle to find an answer to. Any help is appreciated, thanks.

Is bad idea to utilize helper functions on integration tests?

In my job I have a small disagreement on whether we should utilize helper functions for making datasets especially in laravel framework. A sample for the test is:

namespace Tests\MyAppTests;  use PHPUnit\Framework\TestCase; use Carbon\Carbon;  class MyTest extends TestCase {   private function makeATwoHourSpan(Carbon $  now, &$  start, &$  end)   {      $  start = new Carbon($  now);      $  start->modify("-1 hour");       $  end = new Carbon($  now);      $  end->modify("+1 hour");   }    public function testSomething()   {      $  now=Carbon::now();      $  start=null;      $  end=null;       this->makeATwoHourSpan($  now, $  start, $  end);      //Rest Of Test Here   }     public function testSomethingElse()   {      $  now=Carbon::now();      $  start=null;      $  end=null;       this->makeATwoHourSpan($  now, $  start, $  end);      //Rest Of Test Here   } } 

The argument that my supervisor says is that using the makeATwoHourSpan method even though makes the code DRY it does not aid the readability of the test. Also he mentioned that a test should be autonomous and a easy to run as standalone without any helper function except the tools provided from the framework.

So my question is: Should I avoid utilizing “helper” functions especially when I make test data when I make tests, or having a function for data creation makes is the way to go?