Peter met Olivia when she recruited him to help her gain access to his father, who was in a mental institute at the time. Olivia blackmailed him to help her save John Scott. They then became a team investigating fringe events.

Peter who is at first not here voluntarily, slowly integrates into this ‘FBI world’, officially become a FBI consultant.
Peter starts to show his true feelings toward Olivia in “Bound”, when he tells her he ‘cares about her’, he then also helps Olivia when she discovers she has been experimented on as a child.

In the beginning of season 2, after they believe Olivia is dead, Peter stops acting like a consultant and steps up to take control of the investigations. After Olivia wakes up, Peter still acts more seriously toward the cases and grows closer to Olivia, especially when Olivia tries to activate her ability to save hundreds of people. It’s in the episode “Jacksonville”, that they almost share a kiss and in that same episode, when she sees him glimmering as they are about to go out for drinks together, Olivia discovers Peter is not from this universe.

Olivia, as Nina Sharp rightly states, is not prepared to lose him and will agree to keep the secret. When Peter discovers he is not from this universe, he leaves the city without warning and runs without having a specific place to go. In the mean time, Olivia is doing everything she can to find Peter. When Olivia eventually does find him, it’s too late and Walternate, the real father of Peter from the other universe, has found him first. Peter eventually accepts the offer to go home to the other side and leaves the other universe.

After an observer leaves a piece of paper with a drawing of Peter and a machine on it, Olivia does everything she can to go to the other side.
Along with other Cortexiphans kids, she crosses over to the other universe with the main objective of finding Peter, it’s when she finally finds him and admits to him that he belongs with her, that they share their first kiss.
Peter agrees to cross back with Olivia, but doesn’t realize that when he is inside the theater and Olivia is outside, the other universe captures the real Olivia and replaces her with her Alter-self.
It is during the episode “Entrada”, that Peter realizes the Olivia he is with, is not his Olivia. It’s also during that episode that Olivia is able to come back.
In the episode “Marionette”, Peter tells Olivia the truth, that he started seeing the Alt-livia, thinking she was her.

The main point of the relationship of Peter and Olivia is not about whether they will ever be together, but more about whether or not they can survive the turmoil.

The story of : Olivia Dunham | Peter Bishop