Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP - iotune #1

Open
wants to merge 44 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
dd3d9a7
chore(ADOPTERS.md): addition of Civo
AlexsJones Apr 22, 2021
1d5f1f1
Fix log pkg 'with()' function so it is not globally cached on default…
davidvossel Apr 26, 2021
f91f1e9
Unit test exercising logger's custom With() function
davidvossel Apr 26, 2021
5276c62
Quarantine operator test
Apr 27, 2021
0f3874b
remove unused configmap reconciliation code
ashleyschuett Apr 23, 2021
71ba56b
reconcile all required Labels and Annotations on ConfigMap
ashleyschuett Apr 23, 2021
21968b3
refactor reconcile ca configmap
ashleyschuett Apr 23, 2021
a497678
adding SIG label to the operator related e2e tests
enp0s3 Apr 26, 2021
f9b7c97
Add startStrategy to vmi spec
andreyod Apr 11, 2021
b34b500
Assign template and container disk tests to sig-compute
akrejcir Apr 27, 2021
fd39d9e
Cleanup version of sig-storage label
brybacki Apr 28, 2021
5d96451
Add unit and functional tests
andreyod Apr 27, 2021
7976de3
Revert "virt-launcher, Fix flaky VMI report"
oshoval Apr 28, 2021
1dcb1f5
reconcile service resource
ashleyschuett Apr 12, 2021
c78c019
Always use domain create flags
andreyod Apr 28, 2021
137638e
ci, Skip rename test test_id:4659
RamLavi Apr 28, 2021
ce9bfd1
Test unpause after start paused
andreyod Apr 29, 2021
097cfe2
Merge pull request #5520 from ashleyschuett/reconcile-configmap
kubevirt-bot Apr 29, 2021
8d4ed52
Merge pull request #5544 from akrejcir/assign-tests
kubevirt-bot Apr 29, 2021
7025c77
Merge pull request #5557 from oshoval/rev
kubevirt-bot Apr 29, 2021
669ace7
Merge pull request #5459 from ashleyschuett/reconcile-service
kubevirt-bot Apr 29, 2021
65c4daf
Quarantine flaky tests
Apr 29, 2021
9f4f78b
refactor, network: Remove podnicfactory
qinqon Apr 13, 2021
e32da3a
live migration monitor: account time in NanoSeconds
vladikr Apr 29, 2021
76ed1a8
Merge pull request #5536 from davidvossel/fix-log-with
kubevirt-bot Apr 30, 2021
c8226b5
Merge pull request #5541 from fgimenez/quarantine-operator-test
fgimenez Apr 30, 2021
98cc40b
Run bazelisk run //plugins/cmd/uploader:uploader -- -workspace /home/…
kubevirt-bot Apr 30, 2021
bdb31bd
service is continually being patched since sessionAffinity is a defau…
ashleyschuett Apr 30, 2021
fad712e
Merge pull request #5527 from kubevirt-bot/autoupdate
kubevirt-bot Apr 30, 2021
bc64dfa
Mark tests as owned by sig-storage
brybacki Apr 30, 2021
375dc31
Merge pull request #5542 from andreyod/start-paused
kubevirt-bot Apr 30, 2021
44c3a3e
Merge pull request #5567 from fgimenez/quarantine-flaky-tests
kubevirt-bot May 1, 2021
afd21d3
Merge pull request #5575 from ashleyschuett/service-reconcile-fix
kubevirt-bot May 1, 2021
a8da67d
Merge pull request #5534 from enp0s3/tests_add_sig_operator
kubevirt-bot May 1, 2021
9c1a718
reconcile prometheus ServiceMonitor resources
ashleyschuett Mar 26, 2021
9b5b631
reconcile PrometheusRules resouce
ashleyschuett Mar 26, 2021
2b0bf53
add unit tests
ashleyschuett Mar 29, 2021
df50926
Merge pull request #5570 from vladikr/lm_moniotr_nano
kubevirt-bot May 1, 2021
f0c773f
Merge pull request #5551 from RamLavi/remove_test_id_4659
kubevirt-bot May 1, 2021
90975f3
Merge pull request #5344 from ashleyschuett/reconcile-crd-prom
kubevirt-bot May 2, 2021
51479cd
Merge pull request #5062 from qinqon/refactor-remove-pidnicfactory
kubevirt-bot May 2, 2021
9dcf55e
Merge pull request #5552 from brybacki/cleanup-sig-storage-markers
kubevirt-bot May 2, 2021
a3e7c53
Merge pull request #5516 from AlexsJones/master
kubevirt-bot May 3, 2021
69d7cce
Add iotune support
zulh-civo May 3, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,4 @@ hack/builder/manifests/
.bazeldnf/
user.bazelrc
.vscode/

1 change: 1 addition & 0 deletions ADOPTERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,4 @@ This is a likely incomplete list of KubeVirt adopters - end-users and distributo
| H3C | 2019 | We distribute KubeVirt as part of CloudOS to enable VM workloads on Kubernetes at customer sites . Follow the [link](https://www.h3c.com/en/Products_Technology/Enterprise_Products/Cloud_Computing/Cloud_Computing_Products/H3C_CloudOS/H3C_CloudOS_full-stack/) for more product information |
| NVIDIA | 2018 | NVIDIA's latest computing platform is built on open-source projects like Kubernetes and KubeVirt to power products like [GeForce NOW](https://www.nvidia.com/en-us/geforce-now/) with more to come.
| [CoreWeave](https://www.coreweave.com) | 2020 | A Kubernetes native cloud provider with focus on GPUs at scale. KubeVirt allows us to co-locate non-containerizable workloads such as Virtual Desktops next to compute intensive containers executing on bare metal. All orchestrated via the Kubernetes API leveraging the same network policies and persistent volumes for both VM and containerized workloads. |
| [Civo](https://www.civo.com) | 2020 | We are using Kubevirt as part of our stack to enable tenant cluster provisioning within Civo cloud. |
84 changes: 35 additions & 49 deletions WORKSPACE

Large diffs are not rendered by default.

4 changes: 4 additions & 0 deletions api/openapi-spec/swagger.json
Original file line number Diff line number Diff line change
Expand Up @@ -12004,6 +12004,10 @@
"description": "If specified, the VMI will be dispatched by specified scheduler. If not specified, the VMI will be dispatched by default scheduler.",
"type": "string"
},
"startStrategy": {
"description": "StartStrategy can be set to \"Paused\" if Virtual Machine should be started in paused state.",
"type": "string"
},
"subdomain": {
"description": "If specified, the fully qualified vmi hostname will be \"\u003chostname\u003e.\u003csubdomain\u003e.\u003cpod namespace\u003e.svc.\u003ccluster domain\u003e\". If not specified, the vmi will not have a domainname at all. The DNS entry will resolve to the vmi, no matter if the vmi itself can pick up a hostname.",
"type": "string"
Expand Down
5 changes: 3 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ require (
github.com/gogo/protobuf v1.3.2
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
github.com/golang/mock v1.4.4
github.com/golang/protobuf v1.4.3
github.com/golang/protobuf v1.5.0
github.com/google/go-github/v32 v32.0.0
github.com/google/goexpect v0.0.0-20190425035906-112704a48083
github.com/google/gofuzz v1.1.0
Expand Down Expand Up @@ -58,7 +58,8 @@ require (
golang.org/x/net v0.0.0-20201110031124-69a78807bb2b
golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4
golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e
google.golang.org/grpc v1.30.0
google.golang.org/grpc v1.32.0
google.golang.org/protobuf v1.26.0
gopkg.in/cheggaaa/pb.v1 v1.0.28
gopkg.in/yaml.v2 v2.3.0
k8s.io/api v0.20.4
Expand Down
148 changes: 9 additions & 139 deletions go.sum

Large diffs are not rendered by default.

199 changes: 136 additions & 63 deletions pkg/handler-launcher-com/cmd/v1/cmd.pb.go

Large diffs are not rendered by default.

10 changes: 10 additions & 0 deletions pkg/handler-launcher-com/cmd/v1/cmd.proto
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,20 @@ message SMBios {
string family = 5;
}

message DiskIoTune {
uint64 totalBytesSec = 1;
uint64 totalIopsSec = 2;
uint64 readBytesSec = 3;
uint64 writeBytesSec = 4;
uint64 readIopsSec = 5;
uint64 writeIopsSec = 6;
}

message VirtualMachineOptions {
SMBios VirtualMachineSMBios = 1;
uint32 MemBalloonStatsPeriod = 2;
repeated string PreallocatedVolumes = 3;
DiskIoTune DiskIoTune = 4;
}

message VMIRequest {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@ func ValidateVirtualMachineInstanceSpec(field *k8sfield.Path, spec *v1.VirtualMa
causes = append(causes, validateCpuPinning(field, spec)...)
causes = append(causes, validateCPUIsolatorThread(field, spec)...)
causes = append(causes, validateCPUFeaturePolicies(field, spec)...)
causes = append(causes, validateStartStrategy(field, spec)...)

maxNumberOfInterfacesExceeded := len(spec.Domain.Devices.Interfaces) > arrayLenMax
if maxNumberOfInterfacesExceeded {
Expand Down Expand Up @@ -1012,6 +1013,28 @@ func validateRequestLimitOrCoresProvidedOnDedicatedCPUPlacement(field *k8sfield.
return causes
}

func validateStartStrategy(field *k8sfield.Path, spec *v1.VirtualMachineInstanceSpec) (causes []metav1.StatusCause) {
if spec.StartStrategy != nil {
if *spec.StartStrategy != v1.StartStrategyPaused {
causes = append(causes, metav1.StatusCause{
Type: metav1.CauseTypeFieldValueInvalid,
Message: fmt.Sprintf("%s is set with an unrecognized option: %s", field.Child("startStrategy").String(), *spec.StartStrategy),
Field: field.Child("startStrategy").String(),
})
} else if spec.LivenessProbe != nil {
causes = append(causes, metav1.StatusCause{
Type: metav1.CauseTypeFieldValueInvalid,
Message: fmt.Sprintf("either %s or %s should be provided.Pausing VMI with LivenessProbe is not supported",
field.Child("startStrategy").String(),
field.Child("livenessProbe").String(),
),
Field: field.Child("startStrategy").String(),
})
}
}
return causes
}

func validateMemoryRequestsAndLimits(field *k8sfield.Path, spec *v1.VirtualMachineInstanceSpec) (causes []metav1.StatusCause) {
if spec.Domain.Resources.Requests.Memory().Value() > 0 && spec.Domain.Resources.Limits.Memory().Value() > 0 && spec.Domain.Resources.Requests.Memory().Value() != spec.Domain.Resources.Limits.Memory().Value() {
causes = append(causes, metav1.StatusCause{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2134,6 +2134,50 @@ var _ = Describe("Validating VMICreate Admitter", func() {
table.Entry("with DNSPolicy None and nil DNSConfig", k8sv1.DNSNone, interface{}(nil), 1,
[]string{fmt.Sprintf("must provide `dnsConfig` when `dnsPolicy` is %s", k8sv1.DNSNone)}),
)
It("should accept valid start strategy", func() {
vmi := v1.NewMinimalVMI("testvmi")
strategy := v1.StartStrategyPaused
vmi.Spec.StartStrategy = &strategy

causes := ValidateVirtualMachineInstanceSpec(k8sfield.NewPath("fake"), &vmi.Spec, config)
Expect(len(causes)).To(Equal(0))
})
It("should allow no start strategy to be set", func() {
vmi := v1.NewMinimalVMI("testvmi")
vmi.Spec.StartStrategy = nil
causes := ValidateVirtualMachineInstanceSpec(k8sfield.NewPath("fake"), &vmi.Spec, config)
Expect(len(causes)).To(Equal(0))
})
It("should reject invalid start strategy", func() {
vmi := v1.NewMinimalVMI("testvmi")
strategy := v1.StartStrategy("invalid")
vmi.Spec.StartStrategy = &strategy

causes := ValidateVirtualMachineInstanceSpec(k8sfield.NewPath("fake"), &vmi.Spec, config)
Expect(len(causes)).To(Equal(1))
Expect(string(causes[0].Type)).To(Equal("FieldValueInvalid"))
Expect(causes[0].Field).To(Equal("fake.startStrategy"))
Expect(causes[0].Message).To(Equal("fake.startStrategy is set with an unrecognized option: invalid"))
})
It("should reject spec with paused start strategy and LivenessProbe", func() {
vmi := v1.NewMinimalVMI("testvmi")
strategy := v1.StartStrategyPaused
vmi.Spec.StartStrategy = &strategy
vmi.Spec.LivenessProbe = &v1.Probe{
InitialDelaySeconds: 2,
Handler: v1.Handler{
HTTPGet: &k8sv1.HTTPGetAction{Host: "test", Port: intstr.Parse("80")},
},
}
vmi.Spec.Domain.Devices.Interfaces = []v1.Interface{*v1.DefaultBridgeNetworkInterface()}
vmi.Spec.Networks = []v1.Network{*v1.DefaultPodNetwork()}

causes := ValidateVirtualMachineInstanceSpec(k8sfield.NewPath("fake"), &vmi.Spec, config)
Expect(len(causes)).To(Equal(1))
Expect(string(causes[0].Type)).To(Equal("FieldValueInvalid"))
Expect(causes[0].Field).To(Equal("fake.startStrategy"))
Expect(causes[0].Message).To(Equal("either fake.startStrategy or fake.livenessProbe should be provided.Pausing VMI with LivenessProbe is not supported"))
})
})
Context("with cpu pinning", func() {
var vmi *v1.VirtualMachineInstance
Expand Down
14 changes: 0 additions & 14 deletions pkg/virt-handler/cache/cache_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -213,8 +213,6 @@ var _ = Describe("Domain informer", func() {
list = append(list, api.NewMinimalDomain("testvmi1"))

domainManager.EXPECT().ListAllDomains().Return(list, nil)
domainManager.EXPECT().GetGuestOSInfo().Return(api.GuestOSInfo{})
domainManager.EXPECT().InterfacesStatus(list[0].Spec.Devices.Interfaces).Return([]api.InterfaceStatus{})

runCMDServer(wg, socketPath, domainManager, stopChan, nil)

Expand All @@ -240,8 +238,6 @@ var _ = Describe("Domain informer", func() {
list = append(list, api.NewMinimalDomain("testvmi1"))

domainManager.EXPECT().ListAllDomains().Return(list, nil)
domainManager.EXPECT().GetGuestOSInfo().Return(api.GuestOSInfo{})
domainManager.EXPECT().InterfacesStatus(list[0].Spec.Devices.Interfaces).Return([]api.InterfaceStatus{})

err := AddGhostRecord("test1-namespace", "test1", "somefile1", "1234-1")
Expect(err).ToNot(HaveOccurred())
Expand Down Expand Up @@ -270,8 +266,6 @@ var _ = Describe("Domain informer", func() {
list = append(list, domain)

domainManager.EXPECT().ListAllDomains().Return(list, nil)
domainManager.EXPECT().GetGuestOSInfo().Return(api.GuestOSInfo{})
domainManager.EXPECT().InterfacesStatus(list[0].Spec.Devices.Interfaces).Return([]api.InterfaceStatus{})

runCMDServer(wg, socketPath, domainManager, stopChan, nil)

Expand All @@ -290,16 +284,12 @@ var _ = Describe("Domain informer", func() {

domain := api.NewMinimalDomain("test")
domainManager.EXPECT().ListAllDomains().Return([]*api.Domain{domain}, nil)
domainManager.EXPECT().GetGuestOSInfo().Return(api.GuestOSInfo{})
domainManager.EXPECT().InterfacesStatus(domain.Spec.Devices.Interfaces).Return([]api.InterfaceStatus{})
// now prove if we make a change, like adding a label, that the resync
// will pick that change up automatically
newDomain := domain.DeepCopy()
newDomain.ObjectMeta.Labels = make(map[string]string)
newDomain.ObjectMeta.Labels["some-label"] = "some-value"
domainManager.EXPECT().ListAllDomains().Return([]*api.Domain{newDomain}, nil)
domainManager.EXPECT().GetGuestOSInfo().Return(api.GuestOSInfo{Name: "fedora"})
domainManager.EXPECT().InterfacesStatus(newDomain.Spec.Devices.Interfaces).Return([]api.InterfaceStatus{api.InterfaceStatus{Name: "ethx"}})

runCMDServer(wg, socketPath, domainManager, stopChan, nil)

Expand All @@ -324,8 +314,6 @@ var _ = Describe("Domain informer", func() {

Expect(ok).To(BeTrue())
Expect(val).To(Equal("some-value"))
Expect(eventDomain.Status.OSInfo.Name).To(Equal("fedora"))
Expect(eventDomain.Status.Interfaces[0].Name).To(Equal("ethx"))
})

It("should detect expired legacy watchdog file.", func() {
Expand Down Expand Up @@ -448,8 +436,6 @@ var _ = Describe("Domain informer", func() {
list = append(list, domain)

domainManager.EXPECT().ListAllDomains().Return(list, nil)
domainManager.EXPECT().GetGuestOSInfo().Return(api.GuestOSInfo{})
domainManager.EXPECT().InterfacesStatus(list[0].Spec.Devices.Interfaces).Return([]api.InterfaceStatus{})

// This file doesn't have a unix sock server behind it
// verify list still completes regardless
Expand Down
6 changes: 3 additions & 3 deletions pkg/virt-handler/migration-proxy/migration-proxy.go
Original file line number Diff line number Diff line change
Expand Up @@ -339,7 +339,7 @@ func NewSourceProxy(unixSocketPath string, tcpTargetAddress string, serverTLSCon
listenErrChan: make(chan error, 1),
serverTLSConfig: serverTLSConfig,
clientTLSConfig: clientTLSConfig,
logger: log.Log.CustomField("uid", vmiUID).CustomField("listening", filepath.Base(unixSocketPath)).CustomField("outbound", tcpTargetAddress),
logger: log.Log.With("uid", vmiUID).With("listening", filepath.Base(unixSocketPath)).With("outbound", tcpTargetAddress),
}
}

Expand All @@ -356,7 +356,7 @@ func NewTargetProxy(tcpBindAddress string, tcpBindPort int, serverTLSConfig *tls
listenErrChan: make(chan error, 1),
serverTLSConfig: serverTLSConfig,
clientTLSConfig: clientTLSConfig,
logger: log.Log.CustomField("uid", vmiUID).CustomField("outbound", filepath.Base(libvirtdSocketPath)),
logger: log.Log.With("uid", vmiUID).With("outbound", filepath.Base(libvirtdSocketPath)),
}

}
Expand All @@ -382,7 +382,7 @@ func (m *migrationProxy) createTcpListener() error {
// update the random port that was selected
m.tcpBindPort = listener.Addr().(*net.TCPAddr).Port
// Add the listener to the log output once we know the port
m.logger = m.logger.CustomField("listening", fmt.Sprintf("%s:%d", m.tcpBindAddress, m.tcpBindPort))
m.logger = m.logger.With("listening", fmt.Sprintf("%s:%d", m.tcpBindAddress, m.tcpBindPort))
}

m.listener = listener
Expand Down
49 changes: 48 additions & 1 deletion pkg/virt-handler/vm.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ import (
"reflect"
"regexp"
"sort"
"strconv"
"strings"
"sync"
"time"
Expand Down Expand Up @@ -511,7 +512,9 @@ func (d *VirtualMachineController) setPodNetworkPhase1(vmi *v1.VirtualMachineIns
return false, nil
}

err = res.DoNetNS(func() error { return network.SetupPodNetworkPhase1(vmi, pid, d.networkCacheStoreFactory) })
err = res.DoNetNS(func() error {
return network.NewVMNetworkConfigurator(vmi, d.networkCacheStoreFactory).SetupPodNetworkPhase1(pid)
})
if err != nil {
_, critical := err.(*network.CriticalNetworkError)
if critical {
Expand Down Expand Up @@ -2412,6 +2415,49 @@ func (d *VirtualMachineController) vmUpdateHelperDefault(origVMI *v1.VirtualMach
}
}

// Find iotune configurations
iotuneConf := &cmdv1.DiskIoTune{}
for _, v := range vmi.Spec.Volumes {
var name string
source := v.VolumeSource
if source.PersistentVolumeClaim != nil || source.DataVolume != nil {
if source.PersistentVolumeClaim != nil {
name = source.PersistentVolumeClaim.ClaimName
} else {
name = source.DataVolume.Name
}

pvc, err := d.clientset.CoreV1().PersistentVolumeClaims(vmi.Namespace).Get(context.Background(), name, metav1.GetOptions{})
if err != nil {
continue
}

annotations := pvc.ObjectMeta.Annotations
for key, value := range annotations {
annValue, _ := strconv.ParseUint(value, 10, 64)

if strings.Contains(key, "iotune.kubevirt.io/total-bytes-sec") {
iotuneConf.TotalBytesSec = annValue
}
if strings.Contains(key, "iotune.kubevirt.io/total-iops-sec") {
iotuneConf.TotalIopsSec = annValue
}
if strings.Contains(key, "iotune.kubevirt.io/read-bytes-sec") {
iotuneConf.ReadBytesSec = annValue
}
if strings.Contains(key, "iotune.kubevirt.io/write-bytes-sec") {
iotuneConf.WriteBytesSec = annValue
}
if strings.Contains(key, "iotune.kubevirt.io/read-iops-sec") {
iotuneConf.ReadIopsSec = annValue
}
if strings.Contains(key, "iotune.kubevirt.io/write-iops-sec") {
iotuneConf.WriteIopsSec = annValue
}
}
}
}

err = hostdisk.ReplacePVCByHostDisk(vmi, d.clientset)
if err != nil {
return err
Expand Down Expand Up @@ -2466,6 +2512,7 @@ func (d *VirtualMachineController) vmUpdateHelperDefault(origVMI *v1.VirtualMach
},
MemBalloonStatsPeriod: period,
PreallocatedVolumes: preallocatedVolumes,
DiskIoTune: iotuneConf,
}

err = client.SyncVirtualMachine(vmi, options)
Expand Down
22 changes: 0 additions & 22 deletions pkg/virt-launcher/virtwrap/agent-poller/agent_poller.go
Original file line number Diff line number Diff line change
Expand Up @@ -130,28 +130,6 @@ func (s *AsyncAgentStore) GetSysInfo() api.DomainSysInfo {
}
}

// GetInterfaceStatus returns the interfaces Guest Agent reported
func (s *AsyncAgentStore) GetInterfaceStatus() []api.InterfaceStatus {
data, ok := s.store.Load(GET_INTERFACES)
interfacesStatus := []api.InterfaceStatus{}
if ok {
interfacesStatus = data.([]api.InterfaceStatus)
}

return interfacesStatus
}

// GetGuestOSInfo returns the Guest OS version and architecture
func (s *AsyncAgentStore) GetGuestOSInfo() api.GuestOSInfo {
data, ok := s.store.Load(GET_OSINFO)
osinfo := api.GuestOSInfo{}
if ok {
osinfo = data.(api.GuestOSInfo)
}

return osinfo
}

// GetGA returns guest agent record with its version if present
func (s *AsyncAgentStore) GetGA() AgentInfo {
data, ok := s.store.Load(GET_AGENT)
Expand Down
18 changes: 18 additions & 0 deletions pkg/virt-launcher/virtwrap/api/deepcopy_generated.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading