You don't need mask for this. Play with gradients and background:
.box {
--b: 10px; /* border*/
--r: 50px; /* radius */
--c1: red; /* border color */
--c2: #000; /* background color */
width: 98%;
height: 400px;
background:
linear-gradient(90deg,var(--c1) calc(2*var(--b)),#0000 0)
calc(-1*var(--b)) 50%/100% calc(100% - 2*var(--r) + var(--b)) repeat-x,
linear-gradient(var(--c1) calc(2*var(--b)),#0000 0)
50% calc(-1*var(--b))/calc(100% - 2*var(--r) + var(--b)) repeat-y,
radial-gradient(var(--r) at var(--r) var(--r),
#0000 calc(100% - var(--b)),var(--c1) calc(98% - var(--b)) 100%,var(--c2) 101%)
calc(-1*var(--r)) calc(-1*var(--r));
}
<div class="box"></div>
And in case you need use mask, here is another idea:
.box {
--b: 10px; /* border*/
--r: 50px; /* radius */
--c1: red; /* border color */
--c2: #000; /* background color */
width: 98%;
height: 400px;
background:
conic-gradient(from 90deg at var(--b) var(--b),#0000 25%,var(--c1) 0)
0 0/calc(100% - var(--b)) calc(100% - var(--b)),
radial-gradient(var(--r) at var(--r) var(--r),var(--c1) 98%,var(--c2))
calc(-1*var(--r)) calc(-1*var(--r));
--_r: calc(var(--r) - var(--b));
-webkit-mask:
radial-gradient(var(--_r) at var(--_r) var(--_r),#0000 98%,#000)
calc(-1*var(--_r)) calc(-1*var(--_r));
}
<div class="box"></div>
And if you want a transparent background and any kind of border, then use my online generator to get such shape https://css-generators.com/custom-corners/
![Custom corner CSS only](https://cdn.statically.io/img/i.sstatic.net/ztwkz.png)
.box {
position: relative;
height: 300px;
--mask:
radial-gradient(48px at 48px 48px,#0000 calc(98% - 10px),#000 calc(100% - 10px) 98%,#0000) -48px -48px,
linear-gradient(90deg,#000 20px,#0000 0) -10px 50% /100% calc(100% - 96px + 10px) repeat-x,
linear-gradient( #000 20px,#0000 0) 50% -10px/calc(100% - 96px + 10px) 100% repeat-y;
}
.box:before {
content: "";
position: absolute;
inset: 0;
background: linear-gradient(red,blue);
-webkit-mask: var(--mask);
mask: var(--mask);
}
<div class="box"></div>
I am applying the shape to a pseudo-element so you can put content inside it. If you don't need content, apply the CSS to the element